Module: LogStruct::StringScrubber

Extended by:
T::Sig
Defined in:
lib/log_struct/string_scrubber.rb

Overview

StringScrubber is inspired by logstop by @ankane: https://github.com/ankane/logstop Enhancements:

  • Shows which type of data was filtered
  • Includes a hash of each filtered email (HashUtils) for request tracing
  • Uses configuration options from LogStruct.config

Constant Summary collapse

URL_PASSWORD_REGEX =

The password in a URL's user info (postgres://user:password@host, redis://:password@host). The user ends at its first colon; the password runs to the last @ before the host.

%r{(//[^\s/:@]*:)[^\s/]+(@)}
ENCODED_URL_PASSWORD_REGEX =

The same in a URL-encoded URL, like https%3A%2F%2Fuser%3Asecret%40example.com cspell:ignore Fuser Asecret

%r{(%2F%2F(?:(?!%3A|%2F|%40)[^\s/&])*%3A)(?:(?!%2F)[^\s/&])+(%40)}i
URL_PASSWORD_REPLACEMENT = 
'\1[PASSWORD]\2'
CUT_URL_PASSWORD_REGEX =

A URL's user info that runs past what the scrubbers saw of a long string, where the cut can separate the password from the @ that closes it. Everything after the user's colon goes: the password can contain an @ of its own (p@ss).

%r{(//[^\s/:@]*:)[^\s/"'<>]*\z}
CUT_ENCODED_URL_PASSWORD_REGEX = 
%r{(%2F%2F(?:(?!%3A|%2F|%40)[^\s/&])*%3A)(?:(?!%2F)[^\s/&"'<>])*\z}i
CUT_URL_PASSWORD_REPLACEMENT = 
'\1[PASSWORD]'
CUT_BASIC_CREDENTIALS_REGEX =

Basic credentials that run past what the scrubbers saw: a cut token doesn't decode, so it can't be checked

%r{\b(Basic\s+)[A-Za-z0-9+/]+=*\z}i
QUERY_STRING_REGEX =

A URL's query string, for the sensitive params in it

/\?([^\s#"'<>]+)/
SENSITIVE_QUERY_PARAMS =

OAuth authorization codes, which no key pattern would filter

T.let(["code"].freeze, T::Array[String])
QUERY_PARAM_REPLACEMENT = 
"[FILTERED]"
EMAIL_REGEX =

Letters and digits of any script (an accented name, a domain in another alphabet)

/(?<![\p{L}\p{N}._%+-])[\p{L}\p{N}._%+-]+@[\p{L}\p{N}.-]+\.\p{L}{2,}(?![\p{L}\p{N}])/
WHOLE_EMAIL_REGEX =

A whole String that is an email address

/\A#{EMAIL_REGEX}\z/
AUTHORIZATION_CREDENTIALS_REGEX =

The credentials after Bearer or Basic, as in an Authorization header. A Bearer token has at least 16 characters, and Basic credentials must decode to user:password, so "Bearer token" and "Basic plan" stay.

%r{\b(Bearer|Basic)(\s+)([A-Za-z0-9\-._~+/]+=*)}i
AUTHORIZATION_CREDENTIALS_REPLACEMENT = 
"[FILTERED]"
CREDIT_CARD_REGEX =

13 to 19 digits, written together, in fours (4111 1111 1111 1111, with a 19-digit card's last three) or in American Express and Diners Club groups (3782 822463 10005). Numbers that fail the Luhn check are left alone, so most IDs and timestamps are too.

/
  (?<![A-Za-z0-9.-])
  (?:
    [2-6]\d{12,18}
    | [2-6]\d{3}(?<sep>[\ -])\d{4}\k<sep>\d{4}\k<sep>\d{4}(?:\k<sep>\d{3})?
    | 3\d{3}(?<sep2>[\ -])\d{6}\k<sep2>\d{4,5}
  )
  (?![A-Za-z0-9]|-\d)
/x
CREDIT_CARD_REPLACEMENT = 
"[CREDIT_CARD]"
PHONE_REGEX =

US numbers: 555-123-4567, 555 123 4567, 555.123.4567 and (555) 123-4567, with or without +1 or 1

/(?<![\w+])(?:\+?1[\s.-]?)?(?:\(\d{3}\)[\s.-]?|\d{3}[\s.-])\d{3}[\s.-]\d{4}(?!\w)/
PHONE_REPLACEMENT = 
"[PHONE]"
SSN_REGEX =

123-45-6789 and 123 45 6789, but not inside a longer number (9-123-45-6789, 123-45-6789-0)

/(?<!\w|\d-)\d{3}[\s-]\d{2}[\s-]\d{4}(?!\w|-\d)/
SSN_REPLACEMENT = 
"[SSN]"
IP_REGEX = 
/\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/
IPV6_CANDIDATE_REGEX =

Text that could be an IPv6 address (2001:db8::1, ::ffff:192.168.1.1, fe80::1%en0 with its zone). IPAddr decides whether it is one, and it must have a digit, so Ruby constants (Face::Bead) and times (12:30:45) stay.

/(?<![\w:.])(?=[\h:.]*\d)(?:\h{0,4}:){2,7}(?:\d{1,3}(?:\.\d{1,3}){3}|\h{1,4})?(?:%\w+)?(?![\w:])/
IP_REPLACEMENT = 
"[IP]"
MAC_REGEX =

Six pairs of hex digits, separated by colons or by dashes

/\b[0-9a-f]{2}([:-])[0-9a-f]{2}(?:\1[0-9a-f]{2}){4}\b/i
MAC_REPLACEMENT = 
"[MAC]"
IDENTIFIER_REGEX =

Hash keys and Symbols like these are never changed by the scrubbers, so they're skipped

/\A[A-Za-z_][A-Za-z0-9_]*\z/

Class Method Summary collapse

Class Method Details

.scrub(string) ⇒ String

Scrub sensitive information from a string

Parameters:

  • string (String)

Returns:

  • (String) 


90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
 
# File 'lib/log_struct/string_scrubber.rb', line 90

def scrub(string)
  return string if string.empty?

  # The scrubbers match letters of any script, which needs valid UTF-8
  string = valid_utf8(string.to_s).dup
  config = LogStruct.config.filters

  scrub_authorization_credentials!(string) if config.authorization_credentials

  # Sensitive params in URLs' query strings
  scrub_query_params!(string) if config.url_query_params && string.include?("?")

  # Passwords in URLs
  if config.url_passwords
    string.gsub!(URL_PASSWORD_REGEX, URL_PASSWORD_REPLACEMENT)
    string.gsub!(ENCODED_URL_PASSWORD_REGEX, URL_PASSWORD_REPLACEMENT) if string.include?("%")
  end

  # Emails
  if config.email_addresses
    string.gsub!(EMAIL_REGEX) { |email| "[EMAIL:#{value_hash(email)}]" }
  end

  # Credit card numbers
  if config.credit_card_numbers
    string.gsub!(CREDIT_CARD_REGEX) do |number|
      luhn_valid?(number.delete(" -")) ? CREDIT_CARD_REPLACEMENT : number
    end
  end

  # Phone numbers
  string.gsub!(PHONE_REGEX, PHONE_REPLACEMENT) if config.phone_numbers

  # SSNs
  string.gsub!(SSN_REGEX, SSN_REPLACEMENT) if config.ssns

  # IPs: IPv6 first, which can end in an IPv4 address
  if config.ip_addresses
    scrub_ipv6_addresses!(string) if string.include?(":")
    string.gsub!(IP_REGEX, IP_REPLACEMENT)
  end

  # MAC addresses
  string.gsub!(MAC_REGEX, MAC_REPLACEMENT) if config.mac_addresses

  # Custom scrubber
  custom_scrubber = LogStruct.config.string_scrubbing_handler
  string = custom_scrubber.call(string) if !custom_scrubber.nil?

  string
end

.scrub_cut_end(string) ⇒ String

The end of the part of a long string the scrubbers saw (Formatter#scrub_limited_string). They saw only the start of a value that runs past it, so a URL password separated from its closing @ (which URL_PASSWORD_REGEX needs) and Basic credentials cut short (which no longer decode) are replaced, however long they were.

Parameters:

  • string (String)

Returns:

  • (String) 


147
148
149
150
151
152
153
154
155
156
157
 
# File 'lib/log_struct/string_scrubber.rb', line 147

def scrub_cut_end(string)
  filters = LogStruct.config.filters
  if filters.url_passwords
    string = string.sub(CUT_URL_PASSWORD_REGEX, CUT_URL_PASSWORD_REPLACEMENT)
      .sub(CUT_ENCODED_URL_PASSWORD_REGEX, CUT_URL_PASSWORD_REPLACEMENT)
  end
  if filters.authorization_credentials
    string = string.sub(CUT_BASIC_CREDENTIALS_REGEX, "\\1#{AUTHORIZATION_CREDENTIALS_REPLACEMENT}")
  end
  string
end

.scrub_name(name) ⇒ T.untyped

A Hash key or a Symbol, scrubbed like a String unless it's a plain identifier (user_id, createdAt). Returns it unchanged, or the scrubbed String when scrubbing changed it.

Parameters:

  • name (T.untyped)

Returns:

  • (T.untyped) 


187
188
189
190
191
192
193
194
195
 
# File 'lib/log_struct/string_scrubber.rb', line 187

def scrub_name(name)
  return name unless name.is_a?(String) || name.is_a?(Symbol)

  string = name.to_s
  return name if IDENTIFIER_REGEX.match?(string)

  scrubbed = scrub(string)
  (scrubbed == string) ? name : scrubbed
end

.valid_utf8(string) ⇒ String

A String as valid UTF-8: binary and other encodings are converted, and bytes that aren't valid become U+FFFD

Parameters:

  • string (String)

Returns:

  • (String) 


169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/log_struct/string_scrubber.rb', line 169

def valid_utf8(string)
  return string if string.ascii_only? || (string.encoding == Encoding::UTF_8 && string.valid_encoding?)

  utf8 = if string.encoding == Encoding::BINARY || string.encoding == Encoding::US_ASCII
    string.dup.force_encoding(Encoding::UTF_8)
  else
    begin
      string.encode(Encoding::UTF_8)
    rescue EncodingError
      string.dup.force_encoding(Encoding::UTF_8)
    end
  end
  utf8.valid_encoding? ? utf8 : utf8.scrub
end

.value_hash(value) ⇒ String

The hash of a value (HashUtils.hash_value). Email addresses are hashed in lowercase, so an address has one hash however it's written, and the same hash under a filtered key as in a scrubbed string.

Parameters:

  • value (String)

Returns:

  • (String) 


162
163
164
 
# File 'lib/log_struct/string_scrubber.rb', line 162

def value_hash(value)
  HashUtils.hash_value(WHOLE_EMAIL_REGEX.match?(value) ? value.downcase : value)
end

.without_query(url) ⇒ String?

The URL without its query string and fragment (for Referer headers)

Parameters:

  • url (String, nil)

Returns:

  • (String, nil) 


199
200
201
202
203
 
# File 'lib/log_struct/string_scrubber.rb', line 199

def without_query(url)
  return url if url.nil?

  url.split(/[?#]/, 2).first || ""
end