Tuesday, January 26, 2010

Procfs-Process Filesystem

The Linux implementation of /proc also clones that of Plan 9. Under Linux, /proc includes a directory for each running process (including kernel processes) at /proc/PID, containing information about that process, notably including:

  • /proc/PID/cmdline, which contains the command which originally started the process.
  • /proc/PID/cwd, a symlink to the current working directory of the process.
  • /proc/PID/environ, a file containing the names and contents of the environment variables that affect the process.
  • /proc/PID/exe, a symlink to the original executable file, if it still exists (a process may continue running after its original executable has been deleted or replaced).
  • /proc/PID/fd, a directory containing a symbolic link for each open file descriptor.
  • /proc/PID/root, a symlink to the root path as seen by the process. For most processes this will be a link to / unless the process is running in a chroot jail.
  • /proc/PID/status, a file containing basic information about a process including its run state and memory usage.
  • /proc/PID/task, a directory containing hard links to any tasks that have been started by this (i.e.: the parent) process.
  • /proc/PID/maps, the memory map showing which addresses currently visible to that process are mapped to which regions in RAM or to files.

It also includes non-process-related system information, although in the 2.6 kernel much of that information moved to a separate pseudo-file system, sysfs, mounted under /sys:

  • depending on the mode of power management (if at all), either directory, /proc/acpi or /proc/apm, which predate sysfs and contain various bits of information about the state of power management.
  • /proc/bus, containing directories representing various buses on the computer, such as input/PCI/USB. This has been largely superseded by sysfs under /sys/bus which is far more informative.
  • /proc/fb, a list of the available framebuffers
  • /proc/cmdline, giving the boot options passed to the kernel
  • /proc/cpuinfo, containing information about the CPU, such as its vendor (and CPU family, model and model names which should allow users to identify the CPU) and its speed (CPU clockspeed), cache size, number of siblings, cores, and CPU flags. It contains a value called "bogomips", frequently misunderstood as measure CPU-speed like a benchmark, while it doesn't actually measure any sensible (for end-users) value at all. It occurs as a side-effect of kernel timer calibration and yields highly varying values depending on CPU type, even at equal clock speeds.
  • /proc/crypto, a list of available cryptographic modules
  • /proc/devices, a list of character and block devices sorted by device ID but giving the major part of the /dev name too
  • /proc/diskstats, giving some information (including device numbers) for each of the logical disk devices
  • /proc/filesystems, a list of the file systems supported by the kernel at the time of listing
  • /proc/interrupts, /proc/iomem, /proc/ioports and the directory /proc/irq, giving some self-explanatory details about the devices (physical or logical) using the various system resources
  • /proc/meminfo, containing a summary of how the kernel is managing its memory.
  • /proc/modules, one of the most important files in /proc, containing a list of the kernel modules currently loaded . It gives some indication ( not always entirely correct) of dependencies.
  • /proc/mounts, a symlink to self/mounts which contains a list of the currently mounted devices and their mount points (and which file system is in use and what mount options are in use).
  • /proc/net, a directory containing a lot of really useful information about the network stack, in particular nf_conntrack which lists existing network connections (particularly useful for tracking routing when iptables FORWARD is used to redirect network connections).
  • /proc/partitions, a list of the device-numbers, their size and /dev names which the kernel has identified as existing partitions (for example if /dev/sda contains a partition table, then /dev/sda1 and others will appear as available partitions). Note that if a partition isn't listed in this file, then a patched version of losetup is around which can essentially mount the partition and connect /dev/loop[n] devices to the various partitions (though it is not certain if these will then appear in /proc/partitions).
  • /proc/scsi, giving information about any devices connected via a SCSI or RAID controller
  • a symbolic link to the current (traversing) process at /proc/self (i.e. /proc/PID/ where PID is that of the current process).
  • /proc/slabinfo, listing statistics on the caches for frequently-used objects in the Linux kernel
  • /proc/swaps, a list of the active swap partitions, their various sizes and priorities
  • Access to dynamically-configurable kernel options under /proc/sys. Under /proc/sys appear directories representing the areas of kernel, containing readable and writable virtual files.
    For example, a commonly referenced virtual file is /proc/sys/net/ipv4/ip_forward, because it is necessary for routing firewalls or tunnels. The file contains either a '1' or a '0': if it is 1 then the IPv4 stack will forward packets not meant for the local host, if it is 0 then it will not.
  • /proc/sysvipc, containing memory sharing and IPC information.
  • /proc/tty, containing information about the current terminals; /proc/tty/driver looks to be a list of the different types of tty available each of which is a list of those of each type
  • /proc/uptime, the length of time the kernel has been running since boot and spent in idle mode (both in seconds)
  • /proc/version, containing the Linux kernel version, distribution number, gcc version number (used to build the kernel) and any other pertinent information relating to the version of the kernel currently running
  • other files depending on various hardware, module configurations, and changes to the kernel.

The basic utilities that use /proc under Linux come in the procps (/proc processes) package, and only function in conjunction with a mounted /proc.

The procfs plays an important role in moving functionality from kernel space to user space. For example the GNU version of ps operates entirely in user mode, using the procfs to obtain its data.

Sunday, January 24, 2010

File System Hierarchy (FHS)

Directory Description
/ Primary hierarchy root and root directory of the entire file system hierarchy.
/bin/ Essential command binaries that need to be available in single user mode; for all users, e.g., cat, ls, cp.
/boot/ Boot loader files, e.g., kernels, initrd; often a separate partition[8]
/dev/ Essential devices, e.g., /dev/null.
/etc/ Host-specific system-wide configuration files (the name comes from et cetera[9]).
/etc/opt/
Configuration files for /opt/.
/etc/X11/
Configuration files for the X Window System, version 11.
/etc/sgml/
Configuration files for SGML.
/etc/xml/
Configuration files for XML.
/home/ Users' home directories, containing saved files, personal settings, etc.; often a separate partition.
/lib/ Libraries essential for the binaries in /bin/ and /sbin/.
/media/ Mount points for removable media such as CD-ROMs (appeared in FHS-2.3).
/mnt/ Temporarily mounted filesystems.
/opt/ Optional application software packages[10].
/proc/ Virtual filesystem documenting kernel and process status as text files, e.g., uptime, network. In Linux, corresponds to a Procfs mount.
/root/ Home directory for the root user.
/sbin/ Essential system binaries, e.g., init, route, mount.
/srv/ Site-specific data which is served by the system.
/tmp/ Temporary files (see also /var/tmp). Often not preserved between system reboots.
/usr/ Secondary hierarchy for read-only user data; contains the majority of (multi-)user utilities and applications.[11]
/usr/bin/
Non-essential command binaries (not needed in single user mode); for all users.
/usr/include/
Standard include files.
/usr/lib/
Libraries for the binaries in /usr/bin/ and /usr/sbin/.
/usr/sbin/
Non-essential system binaries, e.g., daemons for various network-services.
/usr/share/
Architecture-independent (shared) data.
/usr/src/
Source code, e.g., the kernel source code with its header files.
/usr/X11R6/
X Window System, Version 11, Release 6.
/usr/local/
Tertiary hierarchy for local data, specific to this host. Typically has further subdirectories, e.g., bin/, lib/, share/.[12]
/var/ Variable files—files whose content is expected to continually change during normal operation of the system—such as logs, spool files, and temporary e-mail files. Sometimes a separate partition.
/var/lib/
State information. Persistent data modified by programs as they run, e.g., databases, packaging system metadata, etc.
/var/lock/
Lock files. Files keeping track of resources currently in use.
/var/log/
Log files. Various logs.
/var/mail/
Users' mailboxes.
/var/run/
Information about the running system since last boot, e.g., currently logged-in users and running daemons.
/var/spool/
Spool for tasks waiting to be processed, e.g., print queues and unread mail.
/var/spool/mail/
Deprecated location for users' mailboxes.
/var/tmp/
Temporary files to be preserved between reboots.

Wednesday, January 20, 2010

Some Interesting links i found

Source of Name Generators:

http://www.ruf.rice.edu/~pound/

LinkSwarm Intelligence in bird flocks

http://webjazz.blogspot.com/search/label/ruby
Link
Source Code:http://github.com/iamwilhelm/frock

Python Script for webparser
http://www.daniweb.com/forums/thread161803.html#

http://www.cyberciti.biz/tips/howto-shutdown-linux-box-automatically.html
Link

http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html

http://www.cyberciti.biz/tips/howto-write-system-utilization-data-to-file.html

http://www.cyberciti.biz/tips/how-do-i-find-out-linux-cpu-utilization.html
Link

http://www.cyberciti.biz/tips/linux-strace-command-examples.html
Link

http://www.cyberciti.biz/files/linux-kernel/Documentation/filesystems/proc.txt

How To Write A Name Generator (In Ruby)

Refer: http://www.skorks.com/2009/07/how-to-write-a-name-generator-in-ruby/

Link

Installing And Using SQLite With Ruby On Windows

SQLite is a great little embeddable database engine that’s meant to be self-contained, easy to use and not require configuration. However when I tried to use it in my Ruby code I found that it wasn’t that straight forward getting everything to work, especially if you’re using Ruby on windows. The information on the old WWW was rather sparse. I worked it out eventually, so in the interests of being a helpful web denizen I thought I’d share.

What To Download And Where To Put It

First thing first, before we involve Ruby directly lets download set up everything we need. You need to download two things from the SQLite website (well strictly speaking you only actually NEED one, but we’ll download both since the other one is useful):

  • sqlitedll-3_6_16.zip – you can think of this one as actually being the SQLite database that you will install on your machine
  • sqlite-3_6_16.zip – this one is a command line utility that can be used to administer a SQLite database

Both of those files are located on the downloads page – http://www.sqlite.org/download.html, under the ‘Precompiled Binaries For Windows’ section.

Once you unzip both of those you will have several files, the important ones are:

  • sqlite3.dll – this is the actual database
  • sqlite3.exe – this is the command line utility

You need to take the DLL and put it somewhere, where Ruby would look for libraries when you execute a script, some possible locations would be, your windows system folder or the bin folder of your Ruby installation (Note: as reader Luis Lavena noted in the comments below, the ruby installation bin folder is by far the best place to put the DLL). On my machine it was:

C:\ruby1.8\bin
C:\WINDOWS\system
C:\WINDOWS\system32

You can also put the command line utility in any of those places as well, but you don’t have to as you can use that one from anywhere (if you do put it in another location make sure that location is in the PATH environment variable otherwise you will have to specify the full path to it every time you want to use it).

You’re now ready to install the SQLite Ruby interface gem. You will need to do the following:

gem install sqlite3-ruby

After it completes you will have a gem similar to the following in your gem repository, in my case:

C:\ruby1.8\lib\ruby\gems\1.8\gems\sqlite3-ruby-1.2.5-x86-mswin32

You’re now ready to work with SQLite in your Ruby application. Note that you need to install the ‘sqlite3-ruby’ gem and not the ‘sqlite-ruby’ one, as that is the old one.

If you did not install the DLL correctly you might get an error similar to the following when you try to run your ruby script:

C:/ruby1.8/lib/ruby/1.8/dl/import.rb:29:in `initialize': unknown error (RuntimeError)
from C:/ruby1.8/lib/ruby/1.8/dl/import.rb:29:in `dlopen'
from C:/ruby1.8/lib/ruby/1.8/dl/import.rb:29:in `dlload'
from C:/ruby1.8/lib/ruby/1.8/dl/import.rb:27:in `each'
from C:/ruby1.8/lib/ruby/1.8/dl/import.rb:27:in `dlload'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/driver/dl/api.rb:31
from C:/ruby1.8/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from C:/ruby1.8/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/driver/dl/driver.rb:1
from C:/ruby1.8/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from C:/ruby1.8/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/database.rb:619:in `load_driver'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/database.rb:617:in `each'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/database.rb:617:in `load_driver'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/database.rb:72:in `initialize'
from D:/ruby-projects/search-engine/search-engine/lib/search-engine-main.rb:21:in `new'
from D:/ruby-projects/search-engine/search-engine/lib/search-engine-main.rb:21

This is because the SQLite ruby interface is trying to use a deprecated driver (it’s called DL and it is buggy and doesn’t work) because it can’t find the native one, you need to make sure you put the DLL in the appropriate spot mentioned above. Also note that putting the DLL in the same directory as the script you’re executing does not work and produces a different error possibly because it has a dependency on MSVCRT.DLL.

How To Use SQLite With Ruby

We can now write some ruby code to test that everything is working, as well as learn to use SQLite a little bit.

require 'sqlite3'
database = SQLite3::Database.new( "new.database" )

database.execute( "create table sample_table (id INTEGER PRIMARY KEY, sample_text TEXT, sample_number NUMERIC);" )

database.execute( "insert into sample_table (sample_text,sample_number) values ('Sample Text1', 123)")
database.execute( "insert into sample_table (sample_text,sample_number) values ('Sample Text2', 456)")

rows = database.execute( "select * from sample_table" )

p rows

Running this code will create a new file called ‘new.database’ in the same folder where you ran the script. This is the sqlite database you’ve just created (sqlite stores data as files). To execute any SQL code using sqlite, you simply call the ‘execute’ method on your database instance and provide it some SQL code as a string. This is what we did to create a table and to insert two rows into it. Our little script produces the following output:

[["1", "Sample Text1", "123.0"], ["2", "Sample Text2", "456.0"]]

Which shows that we can get the two rows that we inserted back from the database.

If you attempt to re-run this script, it will simply pick up the database file since the file persists past the execution of the script. Incidentally the script would fail a second time since the table ‘sample_table’ already exists in the database.

You now have the basics down, from here the going is a little easier. You can look at the following resources:

SQLite/Ruby FAQ – all the different ways to use sqlite from a Ruby script

SQLite Documentation – self explanatory

Supported SQL Syntax – self explanatory

SQLite Datatypes – how datatypes work in SQLite 3

Enjoy!

Sunday, January 17, 2010

Learning basics of LaTeX thru Kile!

Ahh....due to some unavoidable forces, this M.tech is teaching me a lot. As a part of it today i'm learning text documentation and usage of LaTeX for that. Here i'm making out some important points that are found useful in the tutorial.

The commands \begin{environment}, \huge and \end{environment} will let you temporarily alter the text inside the environment commands to be size huge.

Changes are only effective from \begin{environment} to \end{environment}. There are no limits as how many changes you can make inside an environment.

Some other features are
  • \begin{emph}: \end{emph}. To italicize one word in a sentence, you can use the syntax: this is \emph{my} sentence
  • \textbf{I am making this text inside the brackets bold}
  • \quote: begin your quote with \begin{quote} and end it with \end{quote}.
  • \center: centers the text.
  • \verse: creates offset text for poems.
  • \itemize: makes an itemized list.
Thats the basic commands i saw and now plunging into kile for creating a new document.