Go ahead!

Memoization for Everything

Download script written in D

| Comments

Tested under dmd 2.058.

This code consists of many D features for scripting :)

Enjoy!

Collect invalid documents for bulk-insert in mongo-ruby-driver

| Comments

Mongo gem 1.6.0 includes my pull request(and HISTORY).

Background

Now, a service consists of many systems in production. As a result, some systems insert broken or invalid data to MongoDB.

Here is problem.

mongo-ruby-driver’s bulk-insert is dead or alive. If inserting docs has one invalid docuemnt, then insert operation failed. In addition, we can’t find invalid documents.

This behavior is not usable. We want to handle invalid documents, e.g. output to local file, ignoring documents and etc.

My pull request resolves this problem.

Usage

I introduced :collect_on_error to insert options.

insert without :collect_on_error:

1
2
# docs is [{}, {}, ...]
result = collection.insert(docs)

result is an array of inserted document.

insert with :collect_on_error:

1
result, invalid_docs = collection.insert(docs, :collect_on_error => true)

result is same as insert without :collect_on_error. invalid_docs is an array of invalid document which removed ObjectId field. We can handle invalid_docs manually. For example, see fluent-plugin-mongo.

Enjoy MongoDB with Ruby!

Released fluent-logger-d

| Comments

fluent-logger-d.

This client library has some TODOs, but works fine. I tested posting 5,000,000 messages with 5 threads, no problem.

In fact, D community now doesn’t want such library. Because current D community doesn’t face to the production.

I hope this library with Fluentd helps your system design in the future.