performance - Speed up InnoDB mysql database import -


i got new mysql machine (16gb ram, 4x sas 15k rpm 300gb drives in raid10), quite fast machine.

i have been importing few databases couple million items each, takes quite long time, longer should think. i'm seeing 1000 inserts per second, , believe should able more. longer process runs, slower gets.

this config, tips speed things up, doesn't hurt reliability of data.

[mysqld] skip-name-resolve datadir=/var/lib/mysql #socket=/tmp/mysql.sock log-error=/var/log/mysqld.log port=48221 user=mysql old_passwords=0 max_connections = 1500 table_cache = 1024 max_allowed_packet = 16m sort_buffer_size = 2m thread_cache = 8 thread_concurrency = 32 query_cache_size = 0m query_cache_type = 0 default-storage-engine = innodb transaction_isolation = repeatable-read tmp_table_size = 256m long_query_time = 3 log_slow_queries = 1 innodb_additional_mem_pool_size=48m innodb_flush_log_at_trx_commit=2 innodb_log_buffer_size=32m innodb_buffer_pool_size=2g innodb_autoinc_lock_mode=2 innodb_io_capacity=500 innodb_read_io_threads=16 innodb_write_io_threads=8 innodb_buffer_pool_size = 5000m innodb_lock_wait_timeout = 300 innodb_max_dirty_pages_pct = 90 innodb_thread_concurrency =32  [mysqldump] quick single-transaction  [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

you drop indexes duration of import, , rebuild them when it's done. should try chunking inserts, searching sweet spot between autocommit , inserts in single transaction.


Comments

Post a Comment

Popular posts from this blog

c++ - Is it possible to compile a VST on linux? -

for class project i'm attempting write vst plugin backed cuda. current cuda workflow on linux box, i'd prefer compile , link there. according wikipedia , should possible (i couldn't find steinberg documentation relevant linux) can't find makefile or instructions on how build if aren't using xcode or visual studio. i'm vst 3 sdk doesn't support linux. when try compile plugin under linux, error: ./base/source/fatomic.cpp:39:30: fatal error: libkern/osatomic.h: no such file or directory this issue caused following code in "vst3 sdk/base/source/fatomic.cpp" #if mac #include <libkern/osatomic.h> #if mac_os_x_version_min_required > mac_os_x_version_10_4 #define native_atomic_type (volatile int32_t*) #else #define native_atomic_type (int32_t*) #endif #elif windows #include <windows.h> #endif but hope compiling under linux work vst sdk 2.4. reading. jvstwrapper seems run on lin...
Read more

java - Output of Eclipse is rubbish -

i want write program in eclipse language different english. when display output using system.out program, displays output correctly in console editor. when accept input console , output it, output rubbish. how can solve it. many thnaks i'm not sure if problem, have character encoding problem. make sure you're using unicode, utf-8. you can read more @ official docs converting non-unicode text .
Read more

jquery - Confused with JSON data and normal data in Django ajax request -

i read json internet still have not got grasp of it. reading article http://webcloud.se/log/ajax-in-django-with-jquery/ i not understood first part function using json def xhr_test(request, format): if request.is_ajax(): if format == 'xml': mimetype = 'application/xml' if format == 'json': mimetype = 'application/javascript' data = serializers.serialize(format, examplemodel.objects.all()) return httpresponse(data,mimetype) # if want prevent non xhr calls else: return httpresponse(status=400) my main problems are from function getting format variable does format json mean data given function json or data recived json can give me simple example ouput of function data = serializers.serialize(format, examplemodel.objects.all()) how use data when response in jquery function if don't use json in above function how input , response chnage thanks ...
Read more