Jaime Frutos Morales's blog

22/11/2009

How to install and configure Sphinx in Ubuntu 9.10 with MySQL support

Filed under: Databases, Ubuntu — acidborg @ 21:40

Description: “Sphinx is a full-text search engine, distributed under GPL version 2. Commercial license is also available for embedded use. Generally, it’s a standalone search engine, meant to provide fast, size-efficient and relevant fulltext search functions to other applications. Sphinx was specially designed to integrate well with SQL databases and scripting languages. Currently built-in data sources support fetching data either via direct connection to MySQL or PostgreSQL, or using XML pipe mechanism (a pipe to indexer in special XML-based format which Sphinx recognizes). As for the name, Sphinx is an acronym which is officially decoded as SQL Phrase Index. Yes, I know about CMU’s Sphinx project“.

Installation:

  • Install the packages needed: apt-get install gcc make libmysqlclient15-dev libmysql++-dev
  • Download Sphinx from here.
  • Decompress it: tar xvfz sphinx-*.tar.gz
  • Enter its directory: cd sphinx-*
  • Run configure: ./configure --prefix=/usr/local/sphinx --with-mysql
  • Compile Sphinx: make
  • Install Sphinx (as root): make install

Configuration:

  • Enter Sphinx’s directory: cd /usr/local/sphinx/etc
  • Make a copy of its default configuration: cp sphinx.conf.dist sphinx.conf
  • Modify it to fit you database schema and preferences (I recommend reading Sphinx’s official documentation and this tutorial by IBM first).
  • Create all indexes: /usr/local/sphinx/bin/indexer --all
  • Something like this will be shown:

    Sphinx 0.9.9-rc2 (r1785)
    Copyright (c) 2001-2009, Andrew Aksyonoff

    using config file '/usr/local/sphinx/etc/sphinx.conf'...
    indexing index 'software'...
    collected 10 docs, 0.0 MB
    sorted 0.0 Mhits, 100.0% done
    total 10 docs, 649 bytes
    total 0.013 sec, 48970 bytes/sec, 754.54 docs/sec
    total 2 reads, 0.000 sec, 16.3 kb/call avg, 0.0 msec/call avg
    total 5 writes, 0.000 sec, 0.4 kb/call avg, 0.0 msec/call avg

Use:

  • You can perform a basic search using: /usr/local/sphinx/bin/search desired_word . It matches all words by default, but you can change this behaviour. Run /usr/local/sphinx/bin/search without arguments to show the available options.
  • Example: /usr/local/sphinx/bin/search samba

    Sphinx 0.9.9-rc2 (r1785)
    Copyright (c) 2001-2009, Andrew Aksyonoff


    using config file '/usr/local/sphinx/etc/sphinx.conf'...
    index 'software': query 'samba ': returned 1 matches of 1 total in 0.000 sec


    displaying matches:
    1. document=10, weight=2
    id=10
    url=http://www.samba.org/
    name=Samba
    description=Samba is a SMB/CIFS file, print, and login server. It seamlessly integrates Linux/Unix Servers and Desktops into Active Directory environments using the Winbind daemon.
    stable_version=3.4.3
    stable_date=2009-10-29
    download=http://www.samba.org/samba/ftp/stable/samba-3.4.3.tar.gz


    words:
    1. 'samba': 1 documents, 2 hits

The use of /usr/local/sphinx/bin/search is just for test and debugging purposes. There are native API ports for PHP, Python, Java, Perl, and Ruby. I will explain the integration between PHP and Sphinx on another post.

20/11/2009

Ubuntu 9.10 CD shipment

Filed under: Ubuntu — acidborg @ 21:14

Today, my official CD of Ubuntu 9.10 (Karmic Koala) arrived. I’d like to thank Canonical for the gift. It will be very useful when installing Ubuntu on friends’ computers because people trust official CDs and that will make my work as a GNU/Linux evangelist easier.

26/10/2009

How to upgrade from Ubuntu 9.04 (Jaunty) to Ubuntu 9.10 (Karmic)

Filed under: SysAdmin, Ubuntu — acidborg @ 15:03
  1. Replace “jaunty” for “karmic” in /etc/apt/sources.list : sed -i 's/jaunty/karmic/g' /etc/apt/sources.list
  2. Run the following commands: apt-get update && apt-get dist-upgrade
  3. Reboot with this command: shutdown -r now

08/10/2009

How to install and configure OSSEC in Ubuntu Server 9.04

Filed under: Security, SysAdmin, Ubuntu — acidborg @ 15:54

Description: “OSSEC is an Open Source Host-based Intrusion Detection System. It performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response. It runs on most operating systems, including Linux, MacOS, Solaris, HP-UX, AIX and Windows.

Installation:

  • Install the needed packages with the following command: apt-get install gcc g++ make
  • Download the latest version from its web.
  • Uncompress it: tar xvfz ossec-hids-latest.tar.gz
  • Run the installation script: cd ossec-hids-* && ./install.sh

Here are the installation steps (with my answers in boldface):



** Para instalação em português, escolha [br].
** 要使用中文进行安装, 请选择 [cn].
** Fur eine deutsche Installation wohlen Sie [de].
** Για εγκατάσταση στα Ελληνικά, επιλέξτε [el].
** For installation in English, choose [en].
** Para instalar en Español , eliga [es].
** Pour une installation en français, choisissez [fr]
** Per l’installazione in Italiano, scegli [it].
** 日本語でインストールします.選択して下さい.[jp].
** Voor installatie in het Nederlands, kies [nl].
** Aby instalować w języku Polskim, wybierz [pl].
** Для инструкций по установке на русском ,введите [ru].
** Za instalaciju na srpskom, izaberi [sr].
** Türkçe kurulum için seçin [tr].
(en/br/cn/de/el/es/fr/it/jp/nl/pl/ru/sr/tr) [en]: en

OSSEC HIDS v2.2 Installation Script – http://www.ossec.net

You are about to start the installation process of the OSSEC HIDS.
You must have a C compiler pre-installed in your system.
If you have any questions or comments, please send an e-mail
to dcid@ossec.net (or daniel.cid@gmail.com).

– System: Linux
– User: root
– Host: server

— Press ENTER to continue or Ctrl-C to abort. —

1- What kind of installation do you want (server, agent, local or help)? local

2- Setting up the installation environment.

– Choose where to install the OSSEC HIDS [/var/ossec]: /opt/ossec

– Installation will be made at /opt/ossec .

3- Configuring the OSSEC HIDS.

3.1- Do you want e-mail notification? (y/n) [y]: y
– What’s your e-mail address? myemail@mydomain.com

– We found your SMTP server as: mail.mydomain.com.
– Do you want to use it? (y/n) [y]: y

— Using SMTP server: mail.mydomain.com.

3.2- Do you want to run the integrity check daemon? (y/n) [y]: y

– Running syscheck (integrity check daemon).

3.3- Do you want to run the rootkit detection engine? (y/n) [y]: y

– Running rootcheck (rootkit detection).

3.4- Active response allows you to execute a specific
command based on the events received. For example,
you can block an IP address or disable access for
a specific user.
More information at:
http://www.ossec.net/en/manual.html#active-response

– Do you want to enable active response? (y/n) [y]: y

– Active response enabled.

– By default, we can enable the host-deny and the
firewall-drop responses. The first one will add
a host to the /etc/hosts.deny and the second one
will block the host on iptables (if linux) or on
ipfilter (if Solaris, FreeBSD or NetBSD).
– They can be used to stop SSHD brute force scans,
portscans and some other forms of attacks. You can
also add them to block on snort events, for example.

– Do you want to enable the firewall-drop response? (y/n) [y]: y

– firewall-drop enabled (local) for levels >= 6

– Default white list for the active response:
– 10.0.0.1
– 10.0.0.2

– Do you want to add more IPs to the white list? (y/n)? [n]: n

3.6- Setting the configuration to analyze the following logs:
— /var/log/messages
— /var/log/auth.log
— /var/log/syslog
— /var/log/mail.info
— /var/log/dpkg.log

– If you want to monitor any other file, just change
the ossec.conf and add a new localfile entry.
Any questions about the configuration can be answered
by visiting us online at http://www.ossec.net .

— Press ENTER to continue —

5- Installing the system
– Running the Makefile
INFO: Little endian set.

(COMPILATION CODE GOES HERE)

– System is Debian (Ubuntu or derivative).
– Init script modified to start OSSEC HIDS during boot.

– Configuration finished properly.

– To start OSSEC HIDS:
/opt/ossec/bin/ossec-control start

– To stop OSSEC HIDS:
/opt/ossec/bin/ossec-control stop

– The configuration can be viewed or modified at /opt/ossec/etc/ossec.conf

Thanks for using the OSSEC HIDS.
If you have any question, suggestion or if you find any bug,
contact us at contact@ossec.net or using our public maillist at
ossec-list@ossec.net
( http://www.ossec.net/main/support/ ).

More information can be found at http://www.ossec.net

— Press ENTER to finish (maybe more information below). —


I chose to perform a local installation because it’s the simplest one and it’s enough for the purpose of this article. I set /opt/ossec as the installation directory according to the Filesystem Hierarchy Standard.

Configuration: You can configure your OSSEC installation editing /opt/ossec/etc/ossec.conf . I recommend to add the email_maxperhour option to limit the number of mails received from OSSEC.

After editing /opt/ossec/etc/ossec.conf , don’t forget to restart OSSEC with this command: /opt/ossec/bin/ossec-control restart .

31/08/2009

How to purge removed DEB packages

Filed under: SysAdmin, Ubuntu — acidborg @ 12:55

When you remove a package in a DEB-based system using apt-get remove package-name (or dpkg -r package-name), if that package has config files, they usually remain in your system.

The best way to remove a package and its config files is using this command: apt-get --purge remove package-name (or dpkg -P package-name).

You can check whether you have removed DEB packages with installed config files using this command: dpkg -l | grep '^rc' (the ‘r’ means ‘removed’ and the ‘c’ means that it has config files installed)

So, if you want to get rid of these config files, just type: dpkg -l | grep '^rc' | awk '{print $2;}' | xargs dpkg --purge

27/08/2009

How to install and configure AppArmor in Ubuntu Server 9.04

Filed under: SysAdmin, Ubuntu — acidborg @ 11:42

Description: “AppArmor (“Application Armor”) is security software for Linux, released under the GNU General Public License. From 2005 through September 2007, AppArmor was maintained by Novell. AppArmor allows the system administrator to associate with each program a security profile that restricts the capabilities of that program. It supplements the traditional Unix discretionary access control (DAC) model by providing mandatory access control (MAC).

In addition to manually specifying profiles, AppArmor includes a learning mode, in which violations of the profile are logged, but not prevented. This log can then be turned into a profile, based on the program’s typical behavior.

AppArmor is implemented using the Linux Security Modules kernel interface.

AppArmor was created in part as an alternative to SELinux, which critics consider difficult for administrators to set up and maintain.[1] Unlike SELinux, which is based on applying labels to files, AppArmor works with file paths. Proponents of AppArmor claim that it is less complex and easier for the average user to learn than SELinux.[2] They also claim that AppArmor requires fewer modifications to work with existing systems:[citation needed] for example, SELinux requires a filesystem that supports “security labels”, and thus cannot provide access control for files mounted via NFS. AppArmor is file-system agnostic.

Installation: Run this command to install it: apt-get install apparmor-profiles apparmor-utils

Configuration (all processes in enforce mode):

  • To show AppArmor’s status use this command: aa-status
  • Output example:

    apparmor module is loaded.
    18 profiles are loaded.
    6 profiles are in enforce mode.
    /usr/lib/NetworkManager/nm-dhcp-client.action
    /usr/sbin/avahi-daemon
    /usr/lib/connman/scripts/dhclient-script
    /usr/sbin/tcpdump
    /sbin/dhclient3
    /sbin/dhclient-script
    12 profiles are in complain mode.
    /usr/sbin/traceroute
    /bin/ping
    /usr/sbin/mdnsd
    /usr/sbin/ntpd
    /usr/sbin/identd
    /usr/sbin/nmbd
    /usr/sbin/dnsmasq
    /sbin/klogd
    /usr/sbin/smbd
    /sbin/syslogd
    /sbin/syslog-ng
    /usr/sbin/nscd
    2 processes have profiles defined.
    0 processes are in enforce mode :
    0 processes are in complain mode.
    2 processes are unconfined but have a profile defined.
    /sbin/klogd (2100)
    /sbin/syslogd (2080)

  • To set enforce mode to all available AppArmor’s profiles use this command: aa-enforce /etc/apparmor.d/*
  • Output example:

    Setting /etc/apparmor.d/bin.ping to enforce mode.
    Setting /etc/apparmor.d/sbin.dhclient3 to enforce mode.
    Setting /etc/apparmor.d/sbin.klogd to enforce mode.
    Setting /etc/apparmor.d/sbin.syslogd to enforce mode.
    Setting /etc/apparmor.d/sbin.syslog-ng to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.avahi-daemon to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.dnsmasq to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.identd to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.mdnsd to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.nmbd to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.nscd to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.ntpd to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.smbd to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.tcpdump to enforce mode.
    Setting /etc/apparmor.d/usr.sbin.traceroute to enforce mode.

  • Execute aa-status again to check the changes:
  • Output example:

    apparmor module is loaded.
    18 profiles are loaded.
    18 profiles are in enforce mode.
    /usr/sbin/traceroute
    /bin/ping
    /usr/sbin/mdnsd
    /usr/lib/NetworkManager/nm-dhcp-client.action
    /usr/sbin/avahi-daemon
    /usr/lib/connman/scripts/dhclient-script
    /usr/sbin/ntpd
    /usr/sbin/identd
    /usr/sbin/tcpdump
    /usr/sbin/nmbd
    /usr/sbin/dnsmasq
    /sbin/klogd
    /usr/sbin/smbd
    /sbin/syslogd
    /sbin/syslog-ng
    /usr/sbin/nscd
    /sbin/dhclient3
    /sbin/dhclient-script
    0 profiles are in complain mode.
    2 processes have profiles defined.
    0 processes are in enforce mode :
    0 processes are in complain mode.
    2 processes are unconfined but have a profile defined.
    /sbin/klogd (2100)
    /sbin/syslogd (2080)

  • Restart processes which are unconfined but have a profile defined. Following my example, these commands should be run:

  • /etc/init.d/klogd restart
    /etc/init.d/sysklogd restart

  • Finally, execute aa-status again and make sure all processes with a defined profile are in enforce mode:
  • Output example:

    apparmor module is loaded.
    18 profiles are loaded.
    18 profiles are in enforce mode.
    /usr/sbin/traceroute
    /bin/ping
    /usr/sbin/mdnsd
    /usr/lib/NetworkManager/nm-dhcp-client.action
    /usr/sbin/avahi-daemon
    /usr/lib/connman/scripts/dhclient-script
    /usr/sbin/ntpd
    /usr/sbin/identd
    /usr/sbin/tcpdump
    /usr/sbin/nmbd
    /usr/sbin/dnsmasq
    /sbin/klogd
    /usr/sbin/smbd
    /sbin/syslogd
    /sbin/syslog-ng
    /usr/sbin/nscd
    /sbin/dhclient3
    /sbin/dhclient-script
    0 profiles are in complain mode.
    2 processes have profiles defined.
    2 processes are in enforce mode :
    /sbin/syslogd (24416)
    /sbin/klogd (24147)
    0 processes are in complain mode.
    0 processes are unconfined but have a profile defined.

24/08/2009

How to install and configure munin server and nodes in Ubuntu Server 9.04

Filed under: SysAdmin, Ubuntu — acidborg @ 09:22

Description: “Munin the monitoring tool surveys all your computers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities. After completing a installation a high number of monitoring plugins will be playing with no more effort.

Using munin you can easily monitor the performance of your computers, networks, SANs, applications, weather measurements and whatever comes to mind. It makes it easy to determine “what’s different today” when a performance problem crops up. It makes it easy to see how you’re doing capacity-wise on any resources.

Munin uses the excellent RRDTool (written by Tobi Oetiker) and the framework is written in Perl, while plugins may be written in any language. munin has a master/node architecture in which the master connects to all the nodes at regular intervals and asks them for data. It then stores the data in RRD files, and (if needed) updates the graphs. One of the main goals has been ease of creating new plugins (graphs).

Due to its client-server architecture, at least one munin server is needed. You can install as many nodes as you want (one per computer or virtual machine you want to monitor).

To install the munin server: apt-get install munin

To configure the munin server, edit /etc/munin/munin.conf and add an entry like the following one for each node (replacing 127.0.0.1 for the IP of the node):

[nodename.domain]
address 127.0.0.1
use_node_name yes

After editing this file, reload your web server to read the changes (usually: /etc/init.d/apache2 restart ).

To install a munin node: apt-get install munin-node

To configure a munin node, edit /etc/munin/munin-node.conf . Use munin as user and group to reduce default permissions:
user munin
group munin

Add a line allowing connections from your munin server’s IP (127.0.0.1 in this example):

allow ^127\.0\.0\.1$

After editing this file, restart your munin node to finish its configuration: /etc/init.d/munin-node restart

24/07/2009

Installing Ntop in Ubuntu Server 9.04

Filed under: SysAdmin, Ubuntu — acidborg @ 12:14

Description: “Ntop is a network traffic probe that shows the network usage, similar to what the popular top Unix command does. Ntop is based on libpcap and it has been written in a portable way in order to virtually run on every Unix platform and on Win32 as well.”

To install it, execute:

apt-get install ntop

Before starting it, configure the interface you want to listen to in /var/lib/ntop/init.cfg

After that, execute manually as root:

ntop

and enter the password for the Ntop’s admin user twice.

I recommend to kill the execution of Ntop using Control+C and start it “the right way” using:

/etc/init.d/ntop start

Finally, you can access Ntop’s web UI using its URL: http://localhost:3000

25/06/2009

Installation & configuration of mod_evasive in Ubuntu Server 9.04

Filed under: SysAdmin, Ubuntu — acidborg @ 14:44

From its README file:

Mod_evasive is an evasive maneuvers module for Apache to provide evasive
action in the event of an HTTP DoS or DDoS attack or brute force attack. It
is also designed to be a detection tool, and can be easily configured to talk
to ipchains, firewalls, routers, and etcetera.

Detection is performed by creating an internal dynamic hash table of IP
Addresses and URIs, and denying any single IP address from any of the following:

  • Requesting the same page more than a few times per second.
  • Making more than 50 concurrent requests on the same child per second.
  • Making any requests while temporarily blacklisted (on a blocking list).”

Its installation is quite easy:

apt-get install libapache2-mod-evasive

To configure it, create a directory to place its log files:

mkdir /var/log/apache2/mod_evasive
chown www-data:www-data /var/log/apache2/mod_evasive

Afterwards, create its configuration file with a default content (change “]” and “[” for “>” and “<" respectively):

[ifmodule mod_evasive20.c]
DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10
DOSLogDir /var/log/apache2/mod_evasive
DOSEmailNotify root@localhost
DOSWhitelist 127.0.0.1
[/ifmodule]

This values should be optimized depending on the use of your server. Here are the params’ descriptions:

  • DOSHashTableSize: Size of the hash table used to store the IPs.
  • DOSPageCount: Number of pages allowed per DOSPageInterval.
  • DOSPageInterval: Time in seconds used by DOSPageCount.
  • DOSSiteCount: Number of objects allowed per DOSSiteInterval.
  • DOSSiteInterval: Time in seconds used by DOSSiteCount.
  • DOSBlockingPeriod: Time in seconds that IPs will be banned. If an IP tries to access the server within this period, the count will be restarted.
  • DOSLogDir: Optional. Directory to store the logs. If not specified, /tmp will be used.
  • DOSEmailNotify: Optional. Mail where notifications will be sent.
  • DOSSystemCommand: Optional. Command to execute if an IP is blocked. For example:
  • DOSSystemCommand "/sbin/iptables -I INPUT -p tcp --dport 80 -s %s -j DROP"

  • DOSWhitelist: Optional. List of IPs which won’t be blocked.
  • To finish the configuration process, restart Apache:

    /etc/init.d/apache2 restart

    You can test whether it works using a script included in the deb package:

    perl /usr/share/doc/libapache2-mod-evasive/examples/test.pl

    23/06/2009

    How to install & use debtags

    Filed under: SysAdmin, Ubuntu — acidborg @ 14:13

    Debtags is a command line interface and maintenance tool for debtags’ information. It’s quite useful to search for deb packages with some specific tags.

    To install it, run the following command:

    apt-get install debtags

    Here are some useful options offered by debtags:

    • Update all packages’ tags:

      debtags update

    • Check debtags’ database:

      debtags selfcheck

    • Show tags’ vocabulary:

      debtags tagcat

    • Show all tags:

      debtags tagcat | grep Tag: | awk {'print $2;'}

    • Show info about a package:

      debtags show package

      Result for debtags show bash:

      Package: bash
      Essential: yes
      Priority: required
      Section: shells
      Installed-Size: 1344
      Maintainer: Ubuntu Core developers
      Original-Maintainer: Matthias Klose
      Architecture: amd64
      Version: 3.2-5ubuntu1
      Replaces: bash-completion (<< 20060301-0), bash-doc (<= 2.05-1) Depends: base-files (>= 2.1.12), debianutils (>= 2.15)
      Pre-Depends: libc6 (>= 2.8), libncurses5 (>= 5.6+20071006-3)
      Recommends: bash-completion (>= 20060301-0)
      Suggests: bash-doc
      Conflicts: bash-completion (<< 20060301-0)
      Filename: pool/main/b/bash/bash_3.2-5ubuntu1_amd64.deb
      Size: 628764
      MD5sum: f71c09143a675a8daede1a668ee98941
      SHA1: 384ef13302e3f11d49399519fe7231c166d253fc
      SHA256: fe15a51dc70b4b0d5ed0556c670ffdf5b0297bb509480f22336684ee156b1d30
      Description: The GNU Bourne Again SHell
      Bash is an sh-compatible command language interpreter that executes
      commands read from the standard input or from a file. Bash also
      incorporates useful features from the Korn and C shells (ksh and csh).
      .
      Bash is ultimately intended to be a conformant implementation of the
      IEEE POSIX Shell and Tools specification (IEEE Working Group 1003.2).
      .
      The Programmable Completion Code, by Ian Macdonald, is now found in
      the bash-completion package.
      Bugs: https://bugs.launchpad.net/ubuntu/+filebug
      Origin: Ubuntu
      Task: minimal, mythbuntu-backend-master, mythbuntu-backend-slave, mythbuntu-desktop, mythbuntu-frontend

    • Show a package’s tags:

      debtags tag ls package

      Result for debtags tag ls bash

      implemented-in::c
      interface::shell
      role::program
      scope::utility
      suite::gnu
      uitoolkit::ncurses

    • Show packets similar or related to a given package:

      debtags related package -d 1

      Result for debtags related bash -d 1

      bash-minimal - The GNU Bourne Again SHell (minimal version)
      es - (short description not available)
      fish - a friendly interactive shell
      tcsh - TENEX C Shell, an enhanced version of Berkeley csh
      zsh-beta - A shell with lots of features (dev tree)

    • More packages are listed if you increment the value of -d.

    • Show packages with a given tag::

      debtags search tag

      For example, to obtain IPv6 enabled packages: debtags search tagprotocol::ipv6

    • Show packages with no tags:

      debtags todo

    • Show stats about packages’ tags:

      debtags stats

      Result for debtags stats in Ubuntu 9.04:

      Total count of packages: 34703
      Total count of packages (according to APT): 34703
      Total count of packages (according to Debtags): 32603
      Number of facets: 30
      Number of tags: 578
      Number of packages with tags, but no special::not-yet-tagged tags: 24547 (75.3%)
      Number of packages with special::not-yet-tagged tags: 8056 (24.7%)
      Number of packages with only special::not-yet-tagged tags: 2932 (9.0%)
      Number of packages with no tags: 0 (0.0%)


    « Newer PostsOlder Posts »

    Create a free website or blog at WordPress.com.