Incorrect string value: '\xCE\xBB, \xCF\x86...' for column 'raw_body' at row 1 about django-silk HOT 6 CLOSED

Not familiar with MySQL, only used postgres and sqlite which explains why I haven't run into this issue myself.

Did a bit of research and doesn't look like Django provides the ability to set charsets on the tables and will assume UTF-8. See here and here for stackoverflow threads where people are facing a similar issue with MySQL.

Looks like you can use the following for database default charsets:

ALTER DATABASE database_name CHARACTER SET utf8 COLLATE utf8_general_ci;

which should ensure that any tables created by Django take UTF from then on. And then if you want to retroactively fit the existing silk tables:

ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8;

Hope this helps.

from django-silk.

Interesting, looks like some kind of encoding issue, see https://docs.djangoproject.com/en/dev/ref/unicode/, in particular the MySQL comments.

I can see from the page content that the charset is utf-8, is your MySQL database configured to support UTF char encodings?

Let me know, if this is the issue i'll add some defensive code to Silk to try and spot this kind of thing. Obviously this error is pretty unhelpful...

Cheers.

from django-silk.

In the meantime if you set the following in settings.py:

SILKY_MAX_REQUEST_BODY_SIZE = 0
SILKY_MAX_RESPONSE_BODY_SIZE = 0

then silk will stop saving the bodies down. As long as you have no UTF characters in the query params/headers etc the error should go away and you can use Silk minus body inspection

from django-silk.

That is a good question, you tell me! :) The encodings are set on a per-table basis.

MySQL says of the silk tables:

mysql> show create table silk_response\G
*************************** 1. row ***************************
       Table: silk_response
Create Table: CREATE TABLE `silk_response` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `request_id` int(11) NOT NULL,
  `status_code` int(11) NOT NULL,
  `raw_body` longtext NOT NULL,
  `body` longtext NOT NULL,
  `encoded_headers` longtext NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `request_id` (`request_id`),
  CONSTRAINT `request_id_refs_id_da922f86` FOREIGN KEY (`request_id`) REFERENCES `silk_request` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

mysql> show create table silk_request\G
*************************** 1. row ***************************
       Table: silk_request
Create Table: CREATE TABLE `silk_request` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `path` varchar(300) NOT NULL,
  `query_params` longtext NOT NULL,
  `raw_body` longtext NOT NULL,
  `body` longtext NOT NULL,
  `method` varchar(10) NOT NULL,
  `start_time` datetime NOT NULL,
  `view_name` varchar(300) NOT NULL,
  `end_time` datetime DEFAULT NULL,
  `time_taken` double DEFAULT NULL,
  `encoded_headers` longtext NOT NULL,
  `meta_time` double DEFAULT NULL,
  `meta_num_queries` int(11) DEFAULT NULL,
  `meta_time_spent_queries` double DEFAULT NULL,
  `num_sql_queries` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `silk_request_73e78684` (`path`),
  KEY `silk_request_b2ae6dce` (`view_name`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

So they're latin-1 by default, it seems.

Thanks for the workaround.

from django-silk.

Closing this as Django does not present any method in which to force table encodings so nothing can be done from the Silk side of things.

from django-silk.

Not familiar with MySQL, only used postgres and sqlite which explains why I haven't run into this issue myself.

Did a bit of research and doesn't look like Django provides the ability to set charsets on the tables and will assume UTF-8. See here and here for stackoverflow threads where people are facing a similar issue with MySQL.

Looks like you can use the following for database default charsets:

ALTER DATABASE database_name CHARACTER SET utf8 COLLATE utf8_general_ci;

which should ensure that any tables created by Django take UTF from then on. And then if you want to retroactively fit the existing silk tables:

ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8;

Hope this helps.

Thanks, this works for me 😄

from django-silk.