Automatic ID assignment in a distributed environment

Just released a new blog post on Zanopia regarding a distributed computing algorithm: The idea is to find a way to assign identifiers to nodes in a distributed cluster without using randomness. The problem explained here.

Accessing S3 buckets with libdroplet

Just released a new blog post on libdroplet, a high performance C library to access Amazon S3 like services:

http://blog.zanopia.org/2010/10/accessing-s3-buckets-with-libdroplet.html

Scality Launches Open Source Cloud Program with $100,000 Incentive Fund for Software Developers

Scality, the company I work for, announced plans to open-source the Software Development Kit (SDK) of its RING technology. As a kickoff incentive, Scality is offering contributing developers bounties from a $100,000 USD fund.

Scality is publishing an open-source library called Scality Droplet, which makes code immediately available for download. Scality Droplet will enable developers to easily build applications which interface with Scality RING, with Amazon’s S3 API and more generally with any object storage technology, thanks to Scality RING’s modular design.

Scality RING is ideal for applications handling enormous volumes of user-generated content such as email in the cloud or social applications, like Facebook, Twitter and YouTube. The Scality Droplet library implements common services such as encryption, compression and large file slicing to let application developers focus on their users’ needs.

The contest or grant opportunity is called the “SCOP Bounty Program,” and it will enable talented developers to apply for $1,000 to $10,000 bounties from a pot of $100,000. To be eligible, the developer’s software application must be built using code freely available from the Scality Droplet library. Interested developers can visit http://scop.scality.com for more information or to download an application form. The submission deadline is November 30, 2010.

ACID transactions on a distributed shared nothing system

New post from me on Zanopia blog:

It is possible to guarantee ACID properties on a scalable database system, with the key notion that reliability is pushed to the storage system, assuming that this storage system is reliable.

http://blog.zanopia.org/2010/09/acid-transactions-can-exist-on-a-shared-nothing-system.html

Evolving data storage models: beyond metadata + data separation

The metadata is the companion data of the data itself. Generally smaller, it contains some important additional information, e.g. (time of last access, access rights, folder which file belongs to, etc).

In a standalone computer system, data and metadata are stored together on a file system on the locally attached hard disk. The metadata is stored generally in the inode of the file.

In a clustered storage system, the current trend is to store data in a storage exclusively dedicated to storage and to store metadata in another cluster dedicated to DB and indexing.


The question is: is it possible to store both metadata and data in the same cluster ?

What are the needs for WebOSes ?

I was just playing with YouOS and wondering what are the needs for such an OS (if we can call this an OS, this is another debate).

1/ One first property is that it can be accessed from any web-2.0-capable device in the same manner: from the office, at home, from a mobile device.

2/ Another important thing is that people don't need complex operating systems any more for simple operations: listening to music, watching videos, browsing the web, chatting (see my previous article on linutops).

WebOSes might be the future of desktop computing. Anyway we have to mention that people won't like their datas to be stored in a remote server: passwords, bank account numbers, illicit mp3's, etc, so we have to find a solution for this.

Analogy between The Google OS and a traditionnal OS

Feature/Concept	Traditional OS	Google OS
CPU		Amazon Elastic Compute Cloud
DMA		Memcached
Interrupts		Google alarms
File management	File system	Distributed file system (e.g. DHT), Amazon Simple Storage Service, Google (secret) distributed filesystem
File search	Agent Ransack	Google search engine
Network protocol	TCP/IP Stack	P2P streaming
Pipes	Unix pipes, named pipes	Yahoo pipes, Microsoft Springfield
Conversation	Unix talk	Google Talk
Web server	Apache	Amazon's Simple Queuing Service
User management	Unix/Windows XP accounts	Google centralized accounts
Window manager	Windows, X Window	Netvibes, Google homepage
Calendar	Outlook	Google calendar
Word processor	MS Office, OpenOffice	Google document
Spread sheet	MS Office, OpenOffice	Google spreadsheet
Presentation	MS Office, OpenOffice	Google presentation