| Deployment GuideChapter 22. Automatic Bug Reporting Tool (ABRT)The Automatic Bug Reporting Tool, commonly abbreviated as ABRT, consists of the abrtd daemon and a number of system services and utilities to process, analyze, and report detected problems. The daemon runs silently in the background most of the time, and springs into action when an application crashes or a kernel oops is detected. The daemon then collects the relevant problem data such as a core file if there is one, the crashing application's command line parameters, and other data of forensic utility. For a brief overview of the most important ABRT components, see Table 22.1, "Basic ABRT components". For Red Hat Enterprise Linux 6.2, the Automatic Bug Reporting Tool has been upgraded to version 2.0. The ABRT 2-series brings major improvements to automatic bug detection and reporting. Table 22.1. Basic ABRT components | Component | Package | Description |
|---|
abrtd | abrt | The ABRT daemon which runs under the root user as a background service. | abrt-applet | abrt-gui | The program that receives messages from abrtd and informs you whenever a new problem occurs. | abrt-gui | abrt-gui | The GUI application that shows collected problem data and allows you to further process it. | abrt-cli | abrt-cli | The command line interface that provides similar functionality to the GUI. | abrt-ccpp | abrt-addon-ccpp | The ABRT service that provides the C/C++ problems analyzer. | abrt-oops | abrt-addon-kerneloops | The ABRT service that provides the kernel oopses analyzer. | abrt-vmcore | abrt-addon-vmcore | The ABRT service that provides the kernel panic analyzer and reporter. |
ABRT currently supports detection of crashes in applications written in the C/C++ and Python languages, as well as kernel oopses. With Red Hat Enterprise Linux 6.3, ABRT can also detect kernel panics if the additional abrt-addon-vmcore package is installed and the kdump crash dumping mechanism is enabled and configured on the system accordingly. ABRT is capable of reporting problems to a remote issue tracker. Reporting can be configured to happen automatically whenever an issue is detected, or problem data can be stored locally, reviewed, reported, and deleted manually by a user. The reporting tools can send problem data to a Bugzilla database, a Red Hat Technical Support (RHTSupport) site, upload it using FTP/SCP, email it, or write it to a file. The part of ABRT which handles already-existing problem data (as opposed to, for example, creation of new problem data) has been factored out into a separate project, libreport. The libreport library provides a generic mechanism for analyzing and reporting problems, and it is used by applications other than ABRT. However, ABRT and libreport operation and configuration is closely integrated. They are therefore discussed as one in this document. Whenever a problem is detected, ABRT compares it with all existing problem data and determines whether that same problem has been recorded. If it has been, the existing problem data is updated and the most recent (duplicate) problem is not recorded again. If this problem is not recognized by ABRT, a problem data directory is created. A problem data directory typically consists of files such as: analyzer, architecture, coredump, cmdline, executable, kernel, os_release, reason, time and uid. Other files, such as backtrace, can be created during analysis depending on which analyzer method is used and its configuration settings. Each of these files holds specific information about the system and the problem itself. For example, the kernel file records the version of the crashed kernel. If you do not use ABRT to further analyze and report the detected problems but instead you report problems using a legacy problem reporting tool, report, note that you can no longer file new bugs. The report utility can now only be used to attach new content to the already existing bugs in the RHTSupport or Bugzilla database. Use the following command to do so: report [[-v]] --target TARGET --ticket ID FILE . . . . . . where TARGET is either strata for reporting to RHTSupport or bugzilla for reporting to Bugzilla. ID stands for number identifying an existing problem case in the respective database, and FILE is a file containing information to be added to the problem case. If you want to report new problems and you do not wish to use abrt-cli, you can now use the report-cli utility instead of report. Issue the following command to let report-cli to guide you through the problem reporting process: report-cli -r DUMP_DIRECTORY . . . . . . where DUMP_DIRECTORY is a problem data directory created by ABRT or some other application using libreport. For more information on report-cli, refer to man report-cli. 22.1. Installing ABRT and Starting its ServicesAs the first step in order to use ABRT, you should ensure that the abrt-desktop package is installed on your system by running the following command as the root user: ~]# yum install abrt-desktop With abrt-desktop installed, you will be able to use ABRT only in its graphical interface. If you intend to use ABRT on the command line, install the abrt-cli package: ~]# yum install abrt-cli Your next step should be to verify that abrtd is running. The daemon is typically configured to start up at boot time. You can use the following command as root to verify its current status: ~]# service abrtd statusabrtd (pid 1535) is running... If the service command returns the abrt is stopped message, the daemon is not running. It can be started for the current session by entering this command: ~]# service abrtd startStarting abrt daemon: [ OK ] Similarly, you can follow the same steps to check and start up the abrt-ccpp service if you want ABRT to catch C/C++ crashes. To set ABRT to detect kernel oopses, use the same steps for the abrt-oops service. Note that this service cannot catch kernel oopses which cause the system to fail, to become unresponsive or to reboot immediately. To be able to detect such kernel oopses with ABRT, you need to install the abrt-vmcore service. If you require this functionality, refer to Section 22.4.5, "Configuring ABRT to Detect a Kernel Panic" for more information. When installing ABRT packages, all respective ABRT services are automatically enabled for runlevels 3 and 5. You can disable or enable any ABRT service for the desired runlevels using the chkconfig utility. Refer to Section 10.2.3, "Using the chkconfig Utility" for more information. Please note that installing ABRT packages overwrites the /proc/sys/kernel/core_pattern file which can contain a template used to name core dump files. The content of this file will be overwritten to: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e Finally, if you run ABRT in a graphical desktop environment, you can verify that the ABRT notification applet is running: ~]$ ps -el | grep abrt-applet0 S 500 2036 1824 0 80 0 - 61604 poll_s ? 00:00:00 abrt-applet If the ABRT notification applet is not running, you can start it manually in your current desktop session by running the abrt-applet program: ~]$ abrt-applet &[1] 2261 The applet can be configured to start automatically when your graphical desktop session starts. You can ensure that the ABRT notification applet is added to the list of programs and selected to run at system startup by selecting the ⤍ ⤍ menu in the top panel. 22.2. Using the Graphical User InterfaceThe ABRT daemon sends a broadcast D-Bus message whenever a problem report is created. If the ABRT notification applet is running, it catches this message and displays an orange alarm icon in the Notification Area. You can open the ABRT GUI application using this icon. As an alternative, you can display the ABRT GUI by selecting the ⤍ ⤍ menu item. Alternatively, you can run the ABRT GUI from the command line as follows: ~]$ abrt-gui & The ABRT GUI provides an easy and intuitive way of viewing, reporting and deleting of reported problems. The ABRT window displays a list of detected problems. Each problem entry consists of the name of the failing application, the reason why the application crashed, and the date of the last occurrence of the problem. If you double-click on a problem report line, you can access the detailed problem description and proceed with the process of determining how the problem should be analyzed, and where it should be reported. You are first asked to provide additional information about the problem which occurred. You should provide detailed information on how the problem happened and what steps should be done in order to reproduce it. In the next steps, choose how the problem will be analyzed and generate a backtrace depending on your configuration. You can skip the analysis and backtrace-generation steps but remember that developers need as much information about the problem as possible. You can always modify the backtrace and remove any sensitive information you do not want to provide before you send the problem data out. Next, choose how you want to report the issue. If you are using Red Hat Enterprise Linux, Red Hat Customer Support is the preferred choice. If you choose to report to Red Hat Customer Support, and you have not configured this event yet, you will be warned that this event is not configured properly and you will be offered an option to do so. After you have chosen a reporting method and have it set up correctly, review the backtrace and confirm the data to be reported. Finally, the problem data is sent to the chosen destination, and you can now decide whether to continue with reporting the problem using another available method or finish your work on this problem. If you have reported your problem to the Red Hat Customer Support database, a problem case is filed in the database. From now on, you will be informed about the problem resolution progress via email you provided during the process of reporting. You can also oversee the problem case using the URL that is provided to you by ABRT GUI when the problem case is created, or via emails received from Red Hat Support. 22.3. Using the Command Line InterfaceProblem data saved by abrtd can be viewed, reported, and deleted using the command line interface. General usage of the abrt-cli tool can be described using the following syntax: abrt-cli [--version] <command> [<args>] . . . . . . where <args> stands for a problem data directory and/or options modifying the commands, and <command> is one of the following sub-commands: list - lists problems and views the problem data.
report - analyzes and reports problems.
rm - removes unneeded problems.
info - provides information about a particular problem.
To display help on particular abrt-cli command use: abrt-cli <command> --help The rest of the commands used with abrt-cli are described in the following sections. To view detected problems, enter the abrt-cli list command: ~]# abrt-cli listDirectory: /var/spool/abrt/ccpp-2011-09-13-10:18:14-2895count: 2executable: /usr/bin/gdbpackage: gdb-7.2-48.el6time: Tue 13 Sep 2011 10:18:14 AM CESTuid: 500Directory: /var/spool/abrt/ccpp-2011-09-21-18:18:07-2841count: 1executable: /bin/bashpackage: bash-4.1.2-8.el6time: Wed 21 Sep 2011 06:18:07 PM CESTuid: 500 Directory - Shows the problem data directory that contains all information about the problem.
count - Shows how many times this particular problem occurred.
executable - Indicates which binary or executable script crashed.
package - Shows the name of the package that contains the program that caused the problem.
time - Shows the date and time of the last occurrence of the problem.
uid - Shows the ID of the user which ran the program that crashed.
The following table shows options available with the abrt-cli list command. All options are mutually inclusive so you can combine them according to your need. The command output will be the most comprehensive if you combine all options, and you will receive the least details if you use no additional options. Table 22.2. The abrt-cli list command options | Option | Description |
|---|
| With no additional option, the abrt-cli list command displays only basic information for problems that have not been reported yet. | -d, --detailed | Displays all stored information about problems listed, including a backtrace if it has already been generated. | -f, --full | Displays basic information for all problems including the already-reported ones. | -v, --verbose | Provides additional information on its actions. |
If you want to view information just about one particular problem, you can use the command: abrt-cli info <directory> . . . . . . where <directory> stands for the problem data directory of the problem that is being viewed. The following table shows options available with the abrt-cli info command. All options are mutually inclusive so you can combine them according to your need. The command output will be the most comprehensive if you combine all options, and you will receive the least details if you use no additional options. Table 22.3. The abrt-cli info command options | Option | Description |
|---|
| With no additional option, the abrt-cli info command displays only basic information for the problem specified by the problem data directory argument. | -d, --detailed | Displays all stored information for the problem specified by the problem data directory argument, including a backtrace if it has already been generated. | -v, --verbose | abrt-cli info provides additional information on its actions. |
22.3.2. Reporting ProblemsTo report a certain problem, use the command: abrt-cli report <directory> ...where <directory> stands for the problem data directory of the problem that is being reported. For example: ~]$ abrt-cli report /var/spool/abrt/ccpp-2011-09-13-10:18:14-2895How you would like to analyze the problem?1) Collect .xsession-errors2) Local GNU DebuggerSelect analyzer: _ ABRT prompts you to select an analyzer event for the problem that is being reported. After selecting an event, the problem is analyzed. This can take a considerable amount of time. When the problem report is ready, abrt-cli opens a text editor with the content of the report. You can see what is being reported, and you can fill in instructions on how to reproduce the crash and other comments. You should also check the backtrace, because the backtrace might be sent to a public server and viewed by anyone, depending on the problem reporter event settings. You can choose which text editor is used to check the reports. abrt-cli uses the editor defined in the ABRT_EDITOR environment variable. If the variable is not defined, it checks the VISUAL and EDITOR variables. If none of these variables is set, vi is used. You can set the preferred editor in your .bashrc configuration file. For example, if you prefer GNU Emacs, add the following line to the file: export VISUAL=emacs When you are done with the report, save your changes and close the editor. You will be asked which of the configured ABRT reporter events you want to use to send the report. How would you like to report the problem?1) Logger2) Red Hat Customer SupportSelect reporter(s): _ After selecting a reporting method, you can proceed with reviewing data to be sent with the report. The following table shows options available with the abrt-cli report command. Table 22.4. The abrt-cli report command options | Option | Description |
|---|
| With no additional option, the abrt-cli report command provides the usual output. | -v, --verbose | abrt-cli report provides additional information on its actions. |
22.3.3. Deleting ProblemsIf you are certain that you do not want to report a particular problem, you can delete it. To delete a problem so ABRT does not keep information about it, use the command: abrt-cli rm <directory> ...where <directory> stands for the problem data directory of the problem being deleted. For example: ~]$ abrt-cli rm /var/spool/abrt/ccpp-2011-09-12-18:37:24-4413rm '/var/spool/abrt/ccpp-2011-09-12-18:37:24-4413' Note that ABRT performs a detection of duplicate problems by comparing new problems with all locally saved problems. For a repeating crash, ABRT requires you to act upon it only once. However, if you delete the crash dump of that problem, the next time this specific problem occurs, ABRT will treat it as a new crash: ABRT will alert you about it, prompt you to fill in a description, and report it. To avoid having ABRT notifying you about a recurring problem, do not delete its problem data. The following table shows options available with the abrt-cli rm command. Table 22.5. The abrt-cli rm command options | Option | Description |
|---|
| With no additional option, the abrt-cli rm command removes the specified problem data directory with all its contents. | -v, --verbose | abrt-cli rm provides additional information on its actions. |
A problem life cycle is driven by events in ABRT. For example: Event 1 - a problem data directory is created. Event 2 - problem data is analyzed. Event 3 - a problem is reported to Bugzilla.
When a problem is detected and its defining data is stored, the problem is processed by running events on the problem's data directory. For more information on events and how to define one, refer to Section 22.4.1, "ABRT Events". Standard ABRT installation currently supports several default events that can be selected and used during problem reporting process. Refer to Section 22.4.2, "Standard ABRT Installation Supported Events" to see the list of these events. Upon installation, ABRT and libreport place their respective configuration files into the several directories on a system: /etc/libreport/ - contains the report_event.conf main configuration file. More information about this configuration file can be found in Section 22.4.1, "ABRT Events". /etc/libreport/events/ - holds files specifying the default setting of predefined events.
/etc/libreport/events.d/ - keeps configuration files defining events.
/etc/libreport/plugins/ - contains configuration files of programs that take part in events.
/etc/abrt/ - holds ABRT specific configuration files used to modify the behavior of ABRT's services and programs. More information about certain specific configuration files can be found in Section 22.4.4, "ABRT Specific Configuration". /etc/abrt/plugins/ - keeps configuration files used to override the default setting of ABRT's services and programs. For more information on some specific configuration files refer to Section 22.4.4, "ABRT Specific Configuration".
Each event is defined by one rule structure in a respective configuration file. The configuration files are typically stored in the /etc/libreport/events.d/ directory. These configuration files are used by the main configuration file, /etc/libreport/report_event.conf. The /etc/libreport/report_event.conf file consists of include directives and rules. Rules are typically stored in other configuration files in the /etc/libreport/events.d/ directory. In the standard installation, the /etc/libreport/report_event.conf file contains only one include directive: include events.d/*.conf
If you would like to modify this file, please note that it respects shell metacharacters (*,$,?, etc.) and interprets relative paths relatively to its location. Each rule starts with a line with a non-space leading character, all subsequent lines starting with the space character or the tab character are considered a part of this rule. Each rule consists of two parts, a condition part and a program part. The condition part contains conditions in one of the following forms: VAR=VAL,
VAR!=VAL, or
VAL~=REGEX
. . . . . . where: VAR is either the EVENT key word or a name of a problem data directory element (such as executable, package, hostname, etc.),
VAL is either a name of an event or a problem data element, and
REGEX is a regular expression.
The program part consists of program names and shell interpretable code. If all conditions in the condition part are valid, the program part is run in the shell. The following is an event example: EVENT=post-create date > /tmp/dt echo $HOSTNAME `uname -r` This event would overwrite the contents of the /tmp/dt file with the current date and time, and print the hostname of the machine and its kernel version on the standard output. Here is an example of a yet more complex event which is actually one of the predefined events. It saves relevant lines from the ~/.xsession-errors file to the problem report for any problem for which the abrt-ccpp services has been used to process that problem, and the crashed application has loaded any X11 libraries at the time of crash: EVENT=analyze_xsession_errors analyzer=CCpp dso_list~=.*/libX11.* test -f ~/.xsession-errors || { echo "No ~/.xsession-errors"; exit 1; } test -r ~/.xsession-errors || { echo "Can't read ~/.xsession-errors"; exit 1; } executable=`cat executable` && base_executable=${executable##*/} && grep -F -e "$base_executable" ~/.xsession-errors | tail -999 >xsession_errors && echo "Element 'xsession_errors' saved"The set of possible events is not hard-set. System administrators can add events according to their need. Currently, the following event names are provided with standard ABRT and libreport installation: -
post-create This event is run by abrtd on newly created problem data directories. When the post-create event is run, abrtd checks whether the UUID identifier of the new problem data matches the UUID of any already existing problem directories. If such a problem directory exists, the new problem data is deleted. -
analyze_<NAME_SUFFIX> . . . . . . where <NAME_SUFFIX> is the adjustable part of the event name. This event is used to process collected data. For example, the analyze_LocalGDB runs the GNU Debugger (GDB) utility on a core dump of an application and produces a backtrace of a program. You can view the list of analyze events and choose from it using abrt-gui. -
collect_<NAME_SUFFIX> . . . . . . where <NAME_SUFFIX> is the adjustable part of the event name. This event is used to collect additional information on a problem. You can view the list of collect events and choose from it using abrt-gui. -
report_<NAME_SUFFIX> . . . . . . where <NAME_SUFFIX> is the adjustable part of the event name. This event is used to report a problem. You can view the list of report events and choose from it using abrt-gui.
Additional information about events (such as their description, names and types of parameters which can be passed to them as environment variables, and other properties) is stored in the /etc/libreport/events/<event_name>.xml files. These files are used by abrt-gui and abrt-cli to make the user interface more friendly. Do not edit these files unless you want to modify the standard installation. 22.4.2. Standard ABRT Installation Supported EventsStandard ABRT installation currently provides a number of default analyzing, collecting and reporting events. Some of these events are also configurable using the ABRT GUI application (for more information on event configuration using ABRT GUI, refer to Section 22.4.3, "Event Configuration in ABRT GUI"). ABRT GUI only shows the event's unique part of the name which is more readable the user, instead of the complete event name. For example, the analyze_xsession_errors event is shown as Collect .xsession-errors in ABRT GUI. The following is a list of default analyzing, collecting and reporting events provided by the standard installation of ABRT: - analyze_VMcore - Analyze VM core
Runs GDB (the GNU debugger) on problem data of an application and generates a backtrace of the kernel. It is defined in the /etc/libreport/events.d/vmcore_event.conf configuration file. - analyze_LocalGDB - Local GNU Debugger
Runs GDB (the GNU debugger) on problem data of an application and generates a backtrace of a program. It is defined in the /etc/libreport/events.d/ccpp_event.conf configuration file. - analyze_xsession_errors - Collect .xsession-errors
Saves relevant lines from the ~/.xsession-errors file to the problem report. It is defined in the /etc/libreport/events.d/ccpp_event.conf configuration file. - report_Logger - Logger
Creates a problem report and saves it to a specified local file. It is defined in the /etc/libreport/events.d/print_event.conf configuration file. - report_RHTSupport - Red Hat Customer Support
Reports problems to the Red Hat Technical Support system. This possibility is intended for users of Red Hat Enterprise Linux. It is defined in the /etc/libreport/events.d/rhtsupport_event.conf configuration file. - report_Mailx - Mailx
Sends a problem report via the Mailx utility to a specified email address. It is defined in the /etc/libreport/events.d/mailx_event.conf configuration file. - report_Kerneloops - Kerneloops.org
Sends a kernel problem to the oops tracker. It is defined in the /etc/libreport/events.d/koops_event.conf configuration file. - report_Uploader - Report uploader
Uploads a tarball (.tar.gz) archive with problem data to the chosen destination using the FTP or the SCP protocol. It is defined in the /etc/libreport/events.d/uploader_event.conf configuration file.
22.4.3. Event Configuration in ABRT GUIEvents can use parameters passed to them as environment variables (for example, the report_Logger event accepts an output file name as a parameter). Using the respective /etc/libreport/events/<event_name>.xml file, ABRT GUI determines which parameters can be specified for a selected event and allows a user to set the values for these parameters. These values are saved by ABRT GUI and reused on subsequent invocations of these events. Open the Event Configuration window by clicking ⤍ . This window shows a list of all available events that can be selected during the reporting process. When you select one of the configurable events, you can click the Configure Event button and you will be able to configure settings for that event. If you change any of the events' parameters, they are saved in the Gnome keyring and will be used in the future GUI sessions. All files in the /etc/libreport/ directory hierarchy are world readable and are meant to be used as global settings. Thus, it is not advisable to store usernames, passwords or any other sensitive data in them. The per-user settings (set in the GUI application and readable by the owner of $HOME only) are stored in the Gnome keyring or can be stored in a text file in $HOME/.abrt/*.conf for use in abrt-cli. The following is a list of all configuration options available for each predefined event that is configurable in the ABRT GUI application. - Logger
In the Logger event configuration window, you can configure the following parameter: When the Append option is checked, the Logger event will append new crash reports to the log file specified in the Logger file option. When unchecked, the new crash report always replaces the previous one. - Red Hat Customer Support
In the Red Hat Customer Support event configuration window, you can configure the following parameters: When the SSL verify option is checked, the SSL protocol is used when sending the data over the network. - MailX
In the MailX event configuration window, you can configure the following parameters: Subject - A string that appears in the Subject field of a problem report email sent by Mailx (by default, set to "[abrt] detected a crash"). Sender - A string that appears in the From field of a problem report email. Recipient - Email address of the recipient of a problem report email.
When the Send Binary Data option is checked, the problem report email will also contain all binary files associated with the problem in an attachment. The core dump file is also sent as an attachment. - Kerneloops.org
In the Kerneloops.org event configuration window, you can configure the following parameter: - Report Uploader
In the Report Uploader event configuration widow, you can configure the following parameter:
22.4.4. ABRT Specific ConfigurationStandard ABRT installation currently provides the following ABRT specific configuration files: /etc/abrt/abrt.conf - allows you to modify the behavior of the abrtd service.
/etc/abrt/abrt-action-save-package-data.conf - allows you to modify the behavior of the abrt-action-save-package-data program.
/etc/abrt/plugins/CCpp.conf - allows you to modify the behavior of ABRT's core catching hook.
The following configuration directives are supported in the /etc/abrt/abrt.conf file: - WatchCrashdumpArchiveDir = /var/spool/abrt-upload
This directive is commented out by default. Enable it if you want abrtd to auto-unpack crashdump tarball archives (.tar.gz) which are located in the specified directory. In the example above, it is the /var/spool/abrt-upload/ directory. Whichever directory you specify in this directive, you must ensure that it exists and it is writable for abrtd. The ABRT daemon will not create it automatically. If you change the default value of this option, be aware that in order to ensure proper functionality of ABRT, this directory must not be the same as the directory specified for the DumpLocation option. Changing the location for crashdump archives will cause SELinux denials unless you reflect the change in respective SELinux rules first. See the abrt_selinux(8) manual page for more information on running ABRT in SELinux. Remember that if you enable this option when using SELinux, you need to execute the following command in order to set the appropriate Boolean allowing ABRT to write into the public_content_rw_t domain: setsebool -P abrt_anon_write 1 - MaxCrashReportsSize =
<size_in_megabytes> This option sets the amount of storage space, in megabytes, used by ABRT to store all problem information from all users. The default setting is 1000 MB. Once the quota specified here has been met, ABRT will continue catching problems, and in order to make room for the new crash dumps, it will delete the oldest and largest ones. - DumpLocation = /var/spool/abrt
This directive is commented out by default. It specifies the location where problem data directories are created and in which problem core dumps and all other problem data are stored. The default location is set to the /var/spool/abrt directory. Whichever directory you specify in this directive, you must ensure that it exists and it is writable for abrtd. If you change the default value of this option, be aware that in order to ensure proper functionality of ABRT, this directory must not be the same as the directory specified for the WatchCrashdumpArchiveDir option. Changing the dump location will cause SELinux denials unless you reflect the change in respective SELinux rules first. See the abrt_selinux(8) manual page for more information on running ABRT in SELinux. Remember that if you enable this option when using SELinux, you need to execute the following command in order to set the appropriate Boolean allowing ABRT to write into the public_content_rw_t domain: setsebool -P abrt_anon_write 1
The following configuration directives are supported in the /etc/abrt/abrt-action-save-package-data.conf file: - OpenGPGCheck =
<yes/no> Setting the OpenGPGCheck directive to yes (the default setting) tells ABRT to only analyze and handle crashes in applications provided by packages which are signed by the GPG keys whose locations are listed in the /etc/abrt/gpg_keys file. Setting OpenGPGCheck to no tells ABRT to catch crashes in all programs. - BlackList = nspluginwrapper, valgrind, strace, [
<MORE_PACKAGES> ] Crashes in packages and binaries listed after the BlackList directive will not be handled by ABRT. If you want ABRT to ignore other packages and binaries, list them here separated by commas. - ProcessUnpackaged =
<yes/no> This directive tells ABRT whether to process crashes in executables that do not belong to any package. The default setting is no. - BlackListedPaths =
/usr/share/doc/*, */example* Crashes in executables in these paths will be ignored by ABRT.
The following configuration directives are supported in the /etc/abrt/plugins/CCpp.conf file: - MakeCompatCore =
<yes/no> This directive specifies whether ABRT's core catching hook should create a core file, as it could be done if ABRT would not be installed. The core file is typically created in the current directory of the crashed program but only if the ulimit -c setting allows it. The directive is set to yes by default. - SaveBinaryImage =
<yes/no> This directive specifies whether ABRT's core catching hook should save a binary image to a core dump. It is useful when debugging crashes which occurred in binaries that were deleted. The default setting is no.
22.4.5. Configuring ABRT to Detect a Kernel PanicWith Red Hat Enterprise Linux 6.3, ABRT can detect a kernel panic using the abrt-vmcore service, which is provided by the abrt-addon-vmcore package. The service starts automatically on system boot and searches for a core dump file in the /var/crash/ directory. If a core dump file is found, abrt-vmcore creates the problem data directory in the /var/spool/abrt/ directory and moves the core dump file to the newly created problem data directory. After the /var/crash/ directory is searched through, the service is stopped until the next system boot. To configure ABRT to detect a kernel panic, perform the following steps: Ensure that the kdump service is enabled on the system. Especially, the amount of memory that is reserved for the kdump kernel has to be set correctly. You can set it by using the system-config-kdump graphical tool, or by specifying the crashkernel parameter in the list of kernel options in the /etc/grub.conf configuration file. See Chapter 26, The kdump Crash Recovery Service for details on how to enable and configure kdump. Install the abrt-addon-vmcore package using the Yum package installer: ~]# yum install abrt-addon-vmcore This installs the abrt-vmcore service with respective support and configuration files. Reboot the system for the changes to take effect.
Unless ABRT is configured differently, problem data for any detected kernel panic is now stored in the /var/spool/abrt/ directory and can be further processed by ABRT just as any other detected kernel oops. 22.4.6. Automatic Downloads and Installation of Debuginfo PackagesABRT can be configured to automatically download and install packages needed for debugging of particular problems. This feature can be useful if you want to debug problems locally in your company environment. To enable automatic debuginfo downloads and installation, ensure that your system fulfills the following conditions: The /etc/libreport/events.d/ccpp_event.conf file contains the following analyzer event, which is present uncommented in default configuration: EVENT=analyze_LocalGDB analyzer=CCpp abrt-action-analyze-core --core=coredump -o build_ids && abrt-action-generate-backtrace && abrt-action-analyze-backtrace The /etc/libreport/events.d/ccpp_event.conf file contains the following line, which allows ABRT to run binary to install debuginfo packages for the problems being analyzed. This line is, in order to avoid installations of unnecessary content, commented out by default so you have to remove the leading # character to enable it: /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache --size_mb=4096 &&
The gdb package, which allows you to generate a backtrace during a problem analysis, is installed on your system. If needed, refer to Section 6.2.4, "Installing Packages" for more information on how to install packages with the Yum package manager.
Note that debuginfo packages are installed using the rhnplugin which requires root privileges. Therefore, you have to run ABRT as root to be able to install debuginfo packages. 22.4.7. Configuring Automatic ReportingABRT can be configured to report any detected issues or crashes automatically without any user interaction. This can be achieved by specifying an analyze-and-report rule as a post-create rule. For example, you can instruct ABRT to report Python crashes to Bugzilla immediately without any user interaction by enabling the rule and replacing the EVENT=report_Bugzilla condition with the EVENT=post-create condition in the /etc/libreport/events.d/python_event.conf file. The new rule will look like the follows: EVENT=post-create analyzer=Python test -f component || abrt-action-save-package-data reporter-bugzilla -c /etc/abrt/plugins/bugzilla.conf Please note that the post-create event is run by abrtd, which usually runs with root privileges. 22.4.8. Uploading and Reporting Using a Proxy ServerThe reporter-bugzilla and the reporter-upload tools respect the http_proxy and the ftp_proxy environment variables. When you use environment variables as a part of a reporting event, they inherit their values from the process which performs reporting, usually abrt-gui or abrt-cli. Therefore, you can specify HTTP or FTP proxy servers by using these variables in your working environment. If you arrange these tools to be a part of the post-create event, they will run as children of the abrtd process. You should either adjust the environment of abrtd or modify the rules to set these variables. For example: EVENT=post-create analyzer=Python test -f component || abrt-action-save-package-data export http_proxy=http://proxy.server:8888/ reporter-bugzilla -c /etc/abrt/plugins/bugzilla.conf 22.5. Configuring Centralized Crash CollectionYou can set up ABRT so that crash reports are collected from multiple systems and sent to a dedicated system for further processing. This is useful when an administrator does not want to log into hundreds of systems and manually check for crashes found by ABRT. In order to use this method, you need to install the libreport-plugin-reportuploader plug-in (yum install libreport-plugin-reportuploader). See the following sections on how to configure systems to use ABRT's centralized crash collection. 22.5.1. Configuration Steps Required on a Dedicated SystemComplete the following steps on a dedicated (server) system: Create a directory to which you want the crash reports to be uploaded to. Usually, /var/spool/abrt-upload/ is used (the rest of the document assumes you are using this directory). Make sure this directory is writable by the abrt user. When the abrt-desktop package is installed, it creates a new system user and a group, both named abrt. This user is used by the abrtd daemon, for example, as the owner:group of /var/spool/abrt/* directories. In the /etc/abrt/abrt.conf configuration file, set the WatchCrashdumpArchiveDir directive to the following: WatchCrashdumpArchiveDir = /var/spool/abrt-upload/ It is advisable to check whether your upload method works. For example, if you use FTP, upload a file using an interactive FTP client: ~]$ ftpftp> open <servername>Name: <username><password>cd /var/spool/abrt-upload250 Operation successfulftp> put <testfile>ftp> quit Check whether <testfile> The MaxCrashReportsSize directive (in the /etc/abrt/abrt.conf configuration file) needs to be set to a larger value if the expected volume of crash data is larger than the default 1000 MB. Consider whether you would like to generate a backtrace of C/C++ crashes. You can disable backtrace generation on the server if you do not wish to generate backtraces at all, or if you decide to create them locally on the machine where a problem occurred. In the standard ABRT installation, a backtrace of a C/C++ crash is generated using the following rule in the /etc/libreport/events.d/ccpp_events.conf configuration file: EVENT=analyze_LocalGDB analyzer=CCpp abrt-action-analyze-core.py --core=coredump -o build_ids && abrt-action-install-debuginfo-to-abrt-cache --size_mb=4096 && abrt-action-generate-backtrace && abrt-action-analyze-backtrace You can ensure that this rule is not applied for uploaded problem data by adding the remote!=1 condition to the rule. Decide whether you want to collect package information (the package and the component elements) in the problem data. Refer to Section 22.5.3, "Saving Package Information" to find out whether you need to collect package information in your centralized crash collection configuration and how to configure it properly.
22.5.2. Configuration Steps Required on a Client SystemComplete the following steps on every client system which will use the central management method: If you decided to not collect package information on client machines, delete, comment out or modify the rule which runs abrt-action-save-package-data in the /etc/libreport/events.d/abrt_event.conf file. Refer to Section 22.5.3, "Saving Package Information" to find out whether you need to collect package information in your centralized crash collection configuration and how to configure it properly. Add a rule for uploading problem reports to the server system in the corresponding configuration file. For example, if you want to upload all problems automatically as soon as they are detected, you can use the following rule in the /etc/libreport/events.d/abrt_event.conf configuration file: EVENT=post-create reporter-upload -u scp://user:[email protected]/directory Alternatively, you can use a similar rule that runs the reporter-upload program as the report_SFX event if you want to store problem data locally on clients and upload it later using ABRT GUI/CLI. The following is an example of such an event: EVENT=report_UploadToMyServer reporter-upload -u scp://user:[email protected]/directory 22.5.3. Saving Package InformationIn a single-machine ABRT installation, problems are usually reported to external bug databases such as RHTSupport or Bugzilla. Reporting to these bug databases usually requires knowledge about the component and package in which the problem occurred. The post-create event runs the abrt-action-save-package-data tool (among other steps) in order to provide this information in the standard ABRT installation. If you are setting up a centralized crash collection system, your requirements may be significantly different. Depending on your needs, you have two options: - Internal analysis of problems
After collecting problem data, you do not need to collect package information if you plan to analyze problems in-house, without reporting them to any external bug databases. You might be also interested in collecting crashes that occur in programs written by your organization or third-party applications installed on your system. If such a program is a part of an RPM package, then on client systems and a dedicated crash collecting system, you can only add the respective GPG key to the /etc/abrt/gpg_keys file or set the following line in the /etc/abrt/abrt-action-save-package-data.conf file: OpenGPGCheck = no If the program does not belong to any RPM package, take the following steps on both, client systems and a dedicated crash collecting system: Remove the following rule from the /etc/libreport/events.d/abrt_event.conf file: EVENT=post-create component= abrt-action-save-package-data Prevent deletion of problem data directories which do not correspond to any installed package by setting the following directive in the /etc/abrt/abrt-action-save-package-data.conf file: ProcessUnpackaged = yes
- Reporting to external bug database
Alternatively, you may want to report crashes to RHTSupport or Bugzilla. In this case, you need to collect package information. Generally, client machines and dedicated crash collecting systems have non-identical sets of installed packages. Therefore, it may happen that problem data uploaded from a client does not correspond to any package installed on the dedicated crash collecting system. In the standard ABRT configuration, this will lead to deletion of problem data (ABRT will consider it to be a crash in an unpackaged executable). To prevent this from happening, it is necessary to modify ABRT's configuration on the dedicated system in the following way: Note that in this case, no such modifications are necessary on client systems: they continue to collect package information, and continue to ignore crashes in unpackaged executables.
22.5.4. Testing ABRT's Crash DetectionAfter completing all the steps of the configuration process, the basic setup is finished. To test that this setup works properly use the kill -s SEGV <PID> command to terminate a process on a client system. For example, start a sleep process and terminate it with the kill command in the following way: ~]$ sleep 100 &[1] 2823~]$ kill -s SEGV 2823 ABRT should detect a crash shortly after executing the kill command. Check that the crash was detected by ABRT on the client system (this can be checked by examining the appropriate syslog file, by running the abrt-cli list --full command, or by examining the crash dump created in the /var/spool/abrt directory), copied to the server system, unpacked on the server system and can be seen and acted upon using abrt-cli or abrt-gui on the server system. |
| |
