linux write data 100% cpu

[little-nil]

Thread 4 (Thread 0x7f6ca5eaa700 (LWP 910)):
#0 0x00007f6cacedc933 in select () from /lib64/libc.so.6
#1 0x00000000004ad6fa in tacopie::io_service::poll() ()
#2 0x00007f6cad77d230 in ?? () from /lib64/libstdc++.so.6
#3 0x00007f6cacbdadf3 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f6cacee51ad in clone () from /lib64/libc.so.6

Threads: 2344 total, 3 running, 2341 sleeping, 0 stopped, 0 zombie
%Cpu(s): 21.0 us, 32.5 sy, 0.0 ni, 46.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem: 16269700 total, 15801636 used, 468064 free, 158872 buffers
KiB Swap: 0 total, 0 used, 0 free. 2272844 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
910 root 20 0 416432 1336 980 R 99.8 0.0 46:30.66 PlayBackServer

[sedapsfognik]

Maybe refers to: #75
Also interesting in this question.

[Cylix]

Hi,

I submitted a possible fix: you can find more information about it in the thread of #75.

Hope it will solve the issue :)

Best!

[LazyPlanet]

linux get also up to 100%

#0 0x00007fbc25bcb6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbc259699ec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x000000000053c20d in std::condition_variable::waitstd::__future_base::_State_base::wait()::{lambda()#1}(std::unique_lockstd::mutex&, std::__future_base::_State_base::wait()::{lambda()#1}) (this=0x7fbc00afb070, __lock=..., __p=...) at /usr/include/c++/4.8.2/condition_variable:93
#3 0x0000000000533500 in std::__future_base::_State_base::wait (this=0x7fbc00afb038) at /usr/include/c++/4.8.2/future:327
#4 0x00000000005449b9 in std::__basic_future<cpp_redis::reply>::_M_get_result (this=0x7fbc1ddb6770) at /usr/include/c++/4.8.2/future:596
#5 0x000000000053c3de in std::future<cpp_redis::reply>::get (this=0x7fbc1ddb6770) at /usr/include/c++/4.8.2/future:675
#6 0x00000000005347e1 in Adoter::Redis::Get (this=0x7fbc00afacb0, key="user:guest_1179", value=..., async=false) at ../Include/RedisManager.h:184

[LazyPlanet]

multithread get data from redis as follow:

bool Get(const std::string& key, std::string& value, bool async = true)
{
if (!_client.is_connected())
{
_client.connect(_hostname, _port);

     if (!_client.is_connected())
     {
         return false;
     }

     auto has_auth = _client.auth(_password);
     if (has_auth.get().ko())
     {
         return false;
     }
 }

 auto get = _client.get(key);
 cpp_redis::reply reply = get.get();

 if (async) {
     _client.commit();
 }
 else {
     _client.sync_commit(std::chrono::milliseconds(1000));
 }

 if (!reply.is_string())
 {
     return false;
 }

 value = reply.as_string();

 return true;

}

thanks.

[Cylix]

Hi @LazyPlanet ,

Are you experiencing high CPU?
Seems like the stacktrace you are showing, the thread is sleeping on a condvar, so it can't be the root cause.
Would you mind sharing a full working small example reproducing the issue? (typically a main that calls your get function and reproduce the 100% CPU).

Thanks

[LazyPlanet]

Most of threads are in that state, It caused by multi-thread, linux centos 7.2 gcc builded, 16 cores, about 1000 calls.
Let me test and tell you in a demo.
Thanks. @Cylix

[sedapsfognik]

@Cylix, after migration to a conditional variable, I'm experiencing 100% CPU very-very rarely. It will be very cool if @LazyPlanet will find how to reproduce this bug.

[LazyPlanet]

The demo , i define a class as follows, when i want a data, i calls Redis().Get(string key);

class Redis
{
private:
std::string _hostname = "127.0.0.1";
int32_t _port = 6379;
std::string _password = "!QAZ%TGBaa.";

cpp_redis::future_client _client;

public:

Redis()
{
std::string hostname = ConfigInstance.GetString("Redis_ServerIP", "127.0.0.1");
int32_t port = ConfigInstance.GetInt("Redis_ServerPort", 6379);
std::string password = ConfigInstance.GetString("Redis_Password", "!QAZ%TGB&UJM9ol.");

	if (port > 0) _port = port;
	if (!hostname.empty()) _hostname = hostname;
	if (!password.empty()) _password = password;
}

bool Get(const std::string& key, std::string& value, bool async = true)
{
if (!_client.is_connected())
{
_client.connect(_hostname, _port);

		if (!_client.is_connected()) 
		{
			return false;
		}
		
		auto has_auth = _client.auth(_password);
		if (has_auth.get().ko()) 
		{
			return false;
		}
	}

	auto get = _client.get(key);
	cpp_redis::reply reply = get.get();
	
	if (async) {
		_client.commit(); 
	} else {
		_client.sync_commit(); 
	}

	if (!reply.is_string()) 
	{
		return false;
	}

	value = reply.as_string();

	return true;
}

bool Save(const std::string& key, const std::string& value, bool async = true)
{
	if (!_client.is_connected()) 
	{
		_client.connect(_hostname, _port);

		if (!_client.is_connected()) 
		{
			return false;
		}
		
		auto has_auth = _client.auth(_password);
		if (has_auth.get().ko()) 
		{
			return false;
		}
	}

	auto set = _client.set(key, value);

	if (async) {
		_client.commit(); 
	} else {
		_client.sync_commit(); 
	}

	auto get = set.get();
	std::string result;
	if (get.is_string()) result = get.as_string();

	return true;
}

};

In multhread i calls 500 times get or save , i will cause as follows:
(gdb) bt
#0 0x00007f257c6776d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f257d2129ec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x000000000053a01b in std::condition_variable::waitstd::__future_base::_State_base::wait()::{lambda()#1}(std::unique_lockstd::mutex&, std::__future_base::_State_base::wait()::{lambda()#1}) (this=0x7f2558005540, __lock=..., __p=...) at /usr/include/c++/4.8.2/condition_variable:93
#3 0x000000000053341a in std::__future_base::_State_base::wait (this=0x7f2558005508) at /usr/include/c++/4.8.2/future:327
#4 0x00000000005414c3 in std::__basic_future<cpp_redis::reply>::_M_get_result (this=0x7f25754440c0) at /usr/include/c++/4.8.2/future:596
#5 0x000000000053a1de in std::future<cpp_redis::reply>::get (this=0x7f25754440c0) at /usr/include/c++/4.8.2/future:675
#6 0x0000000000533c77 in Adoter::Redis::Get (this=0x7f2558004880, key="player:2101280", value=..., async=false) at ../Include/RedisManager.h:186
#7 0x000000000053477e in Adoter::Redis::GetPlayer (this=0x7f2558004880, player_id=2101280, player=...) at ../Include/RedisManager.h:313

Do I misuse it?

[Cylix]

Hi,

You should call commit or sync_commit before calling .get() on the std::future object returns by the redis client.
Right now, you are waiting indefinitely while you haven't pushed the pipelined commands to the redis server.

Remember, when you call .get, .set or any other redis command, the command is not sent yet to the redis server. Instead it is buffered and flushed when you can commit or sync_commit.

[LazyPlanet]

Oh, sorry!!
I am wrong!!
Thank u very much, I have thought it is because I donnot put paras timeout in sync_commit. such as _client.sync_commit(std::chrono::milliseconds(100));

[LazyPlanet]

I am sorry to interrupt you that there is a lock, Is there I am wrong? About 100 connections with redis at this time.
Thank you very much.

void Game::SavePlayBack()
{
if (!_room) return;

cpp_redis::future_client client;
client.connect(ConfigInstance.GetString("Redis_ServerIP", "127.0.0.1"), ConfigInstance.GetInt("Redis_ServerPort", 6379));
if (!client.is_connected()) return;

auto has_auth = client.auth(ConfigInstance.GetString("Redis_Password", "!ssl."));
if (has_auth.get().ko()) return; ///error

auto set = client.set("playback:" + std::to_string(_room_id) + "_" + std::to_string(_game_id), _playback.SerializeAsString());
client.commit();

auto get = set.get();
std::string result;
if (get.is_string()) result = get.as_string();

//auto redis_cli = make_unique<Redis>();
//std::string key = "playback:" + std::to_string(_room_id) + "_" + std::to_string(_game_id);
//redis_cli->Save(key, _playback);  

}
#0 0x00007f257c6776d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f257d2129ec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x000000000053a01b in std::condition_variable::waitstd::__future_base::_State_base::wait()::{lambda()#1}(std::unique_lockstd::mutex&, std::__future_base::_State_base::wait()::{lambda()#1}) (this=0x7f2558005540, __lock=..., __p=...) at /usr/include/c++/4.8.2/condition_variable:93
#3 0x000000000053341a in std::__future_base::_State_base::wait (this=0x7f2558005508) at /usr/include/c++/4.8.2/future:327
#4 0x00000000005414c3 in std::__basic_future<cpp_redis::reply>::_M_get_result (this=0x7f25754440c0) at /usr/include/c++/4.8.2/future:596
#5 0x000000000053a1de in std::future<cpp_redis::reply>::get (this=0x7f25754440c0) at /usr/include/c++/4.8.2/future:675

[LazyPlanet]

I want to know if I use _client.sync_commit(std::chrono::milliseconds(100)); , can it not cause pthread_cond_wait ?
Thank u.