HOWTO Python Setup UNICODE Logging

Overview

Python does a very good job handling UNICODE for most modules, however, logging is not one of then. To get around this is easy but a special technique is required.

Common Approach

Normally with logging you pass a file to the logger and then write to it as you need though the logger interface. This approach is fast but does not allow the setting of the file encoding.

import logging
logging.basicConfig(
    level=logging.DEBUG,
    filename=filename,
    filemode="w")

formatter = logging.Formatter('%(asctime)-15s: %(levelname)-7s - %(message)s')
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)

log = logging.getLogger("MAIN")
log.addHandler(ch)
_logger = log

Improved Approach

To get around this and to ensure you control the formatting, a file stream will need to be created first and then that stream can be passed to logger.

import logging
import logging.handlers
log_fh = open("/logs/my_unicode.log", "w", encoding="utf-8")
logging.basicConfig(
    level=logging.DEBUG,
    format=config.FORMAT)

formatter = logging.Formatter('%(asctime)-15s: %(levelname)-7s - %(message)s')
ch = logging.StreamHandler(log_fh)
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)

log = logging.getLogger("MAIN")
log.addHandler(ch)

The log_fh holds the handle to the UTF–8 stream which is passed to the logger StreamHandler. Once set, text will be properly encoded as it is passed to the logger.

Conclusion

Using a slightly altered technique it should be possible to write anything to a python log using the build in logger module. There is no reason to use this technique for all logging even if you do not think UNICODE will be used.