CleanSpeak Release Notes

Version 3.33.0

Released Thursday December 28th, 2023

Changed

  • If you are using MySQL, the installation procedure has changed. Please review the installation guide to ensure you are able to download the MySQL JDBC connector as part of your installation process. See Install MySQL Connector in the Database installation guide for additional details. If you need further assistance, please open a support ticket with CleanSpeak support.

Fixed

  • When a phrase match is also upgraded to a username match, a filter request may return a 500 status code with an exception in the logs.

Internal

Version 3.32.5

Released Sunday September 10th, 2023

Fixed

  • The Finnish inflector may fail when you configure a filter entry as Finnish and the word is entirely multi-byte unicode characters instead of ASCII.

Version 3.32.4

Released Tuesday June 20th, 2023

Fixed

  • The filter may not reload after approving a blacklist phrase.

Version 3.32.3

Released Friday July 8th, 2022

Fixed

  • The weekly report PDF may fail to render if the content contains unicode characters because unicode cannot be rendered by the PDF fonts.

  • Fix up a few remaining uses of old email addresses or web addresses in CleanSpeak.

  • Support Elasticsearch URLs containing credentials in the user-information portion of a URL that contain URL percent encoded :. This is the completed fix for the issue identified and partially resolved in 3.32.2 under GitHub Issue #66.

Version 3.32.2

Released Wednesday June 8th, 2022

Fixed

  • Improve validation when importing a blacklist in the UI that contains a malformed locale string.

  • Fix PDF extension when selecting the download report. Some browsers will not correctly infer the correct PDF extension, so the file may not automatically open with a PDF viewer.

  • Support Elasticsearch URLs containing credentials in the user-information portion of a URL that contain URL percent encoding.

Internal

Version 3.32.1

Released Friday February 18th, 2022

Security

  • Proactively upgrade Logback. Instead of Log4J, CleanSpeak uses Logback. In response to the recent vulnerabilities in Log4J, the Logback team has proactively added some additional hardening to their library to ensure similar vulnerabilities are not found.

Fixed

  • Some OpenID Connect IdPs such as Okta strictly enforce only sending the client_id in the request body or the Authorization header. This fix is to only send the client_id in the request body when you request to use client_secret_post or none for the Client authentication method. When configuring OpenID Connect in the CleanSpeak UI, you may now select the client authentication method you wish to use with the OpenID Connect identity provider.

Enhancement

  • Add the ability to override or provide additional custom SMTP options. This should allow CleanSpeak to be more compatible with 3rd party SMTP services.

Version 3.32.0

Released Friday January 21st, 2022

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Security

  • Upgraded Logback to the latest available version of 1.3.0.

    • No known vulnerabilities exist in the version previously shipping with CleanSpeak. As a resul of the Log4J vulnerability, the Logback team added some additional hardening to their library as a precautionary measure. We are upgrading to the latest version that includes these additions.

    • Resolves GitHub Issue #49

  • Add -Dlog4j2.formatMsgNoLookups=true to the VM options shipped with the cleanspeak-search-engine Elasticsearch package.

Fixed

  • Selecting a custom duration for the combined Zip download in the admin UI may fail.

Internal

  • CleanSpeak now shipping with 21% more Java!

  • Upgraded PostgreSQL JDBC driver from 4.2.22 to 42.3.1

  • Update prime-email dependency.

Version 3.31.4

Released Thursday November 25th, 2021

Fixed

  • Moderate escalation comments not displayed in the escalation queue.

  • Unable to perform an administrative score change on a content user from the User Moderation queue.

    • This occurred when managing user at the URI /admin/moderation/user.

    • Resolves GitHub Issue #43

  • Several fixes for the Zip Report feature released in 3.31.3.

Version 3.31.3

Released Tuesday November 9th, 2021

Fixed

  • When adding an automated user action for a user action that is not time based you are unable to save.

  • Restoring from backup when the array WhitelistedURLs has an entry that is a String instead of an Object throws an exception.

  • Editing a blacklist phrase and approving the change does not show in the audit log.

  • Change the default image filtering probability for Low matches from 0 to 30.

  • If an event has never been configured (enabled or disabled) for a webhook an exception may occur when that event occurs, and an attempt to send that event to the webhook occurs.

  • Attempting to action a user from the dashboard, and many other places is blocked by the fact that selecting an action does not update the rest of the UI. Specifically, the related controls are not properly hidden or shown, and the submit button is never enabled.

  • When submitting an escalation comment that is longer than 255 characters saving the escalation will fail.

  • When adding a moderation action with any one of location, sender display name, or receiver display name that is longer than 255 characters it will block them from being asynchronously added to the database. The action gets stuck in the queue and is never cleared or saved.

  • Fix the queue drainer so that it is properly closed during shutdown. Outstanding queue entries are written to disk, so they can be replayed later.

  • The OIDC integration generates invalid OAuth redirect URLs when starting the OAuth login process. The URLs have two ? symbols, which make them invalid.

  • The css text color for h3 inside certain tooltips is not correct. It’s black on black, so you can’t read the text.

Enhancements

  • Add an option to the system configuration to use a proxy for all outbound network access to CleanSpeak licensing.

  • Support vv as a leet speak replacement for w.

  • Added a new reports UI to download a zip file of several reports: content, filtered entries, filtered words, moderator activity, top filtered users, and top producing users. The new report can be found in the UI by navigating to Reports → Combined Zip Report.

  • Add a queue size to the queue drainer health check.

Internal

Version 3.31.2

Released Monday August 17th, 2021

Fixed

  • A missing JDK module in the docker image may cause a runtime failure. This fix only affects version 3.31.1.

Version 3.31.1

Released Monday August 16th, 2021

Fixed

  • Updated docker images to use new build strategy and correct the user owning the CleanSpeak process.

Version 3.31.0

Released Wednesday June 16th, 2021

Fixed

  • When editing a webhook in the CleanSpeak admin UI, you may lose your webhook configuration during the save operation.

Version 3.30.0

Released Thursday May 13th, 2021

Please Read

If you are upgrading from a version of CleanSpeak older than 3.4.0, you will need to reconfigure your SMTP settings in the Management Interface UI in order for emails to be sent by CleanSpeak.

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Changed

  • Removed System V init.d scripts from the RPM and DEB packages and replaced them with SystemD service definitions.

  • Upgraded to Java 14.

  • Upgraded to support Elasticsearch 7.x (version 6.x is still supported as well).

  • Upgraded all dependencies for security fixes and improvements.

  • Java is no longer bundled with the Zip, RPM or Debian packages. Java will be downloaded during the initial startup.

  • When using RPM or Debian packages, the installation path has been updated to /usr/local/cleanspeak from /usr/local/inversoft. When upgrading, the cleanspeak.properties file will be migrated to the new location.

Fixed

  • A JavaScript bug that may cause an error on the Approval Queue and the Content Alert queue has been resolved.

  • Error when deleting Applications that had moderators working in queues previously.

  • Pagination fixed for Application listing page.

Enhancement

  • When under heavy load with increased database latency, it is possible for the User score processing services to cause additional database contention. This has been enhanced to perform some of the score processing during the request and short circuit some asynchronous processing that may require additional database locks.

Version 3.29.0-RC.1

Released Thursday, April 16th, 2020

New

  • Proxy endpoint for allowing moderate responses to instead be forwarded downstream to a different server than the requester

  • Proxy configuration in the application moderation configuration. When you use the moderate proxy your application requires a proxy to be configured. These settings allow you to choose how to forward your moderate results.

  • Webhooks section in the application settings. It was previously tedious to go to the webhooks page to select all the applications to apply a given webhook to. You can now directly enable webhooks from the application settings.

Version 3.28.1

Released Thursday, March 12th, 2020

Please Read

This release removes the SightEngine and Rekognition integrations. If you are using these features then it is not recommended to upgrade without considering the required code changes with your integration. These media features are now directly integrated with CleanSpeak for improved performance and capability. Use of these new media filters are available for an additional cost.

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Changed

  • Removed the SightEngine and Rekognition options from the system and application configurations

  • Removed video content type from verification tool. Video is now always asynchronous so it is not possible to test in the UI anymore.

New

  • Added Offensive, Nudity, Scam, and the [Weapon/Alcohol/Drug] filters for both image and video filtering

    • Offensive can detect offensive content like swastikas, SS bolts, and middle fingers

    • Nudity can detect x-rated content as well as suggestive content

    • Scam detects images that are frequently associated with scammers

    • Weapon/Alcohol/Drug filter detects the presence of weapons, alcohol, or drugs

  • Added media filter configuration to every application

    • Previously media filters were configured globally on the system configuration and now they are configured per application with far more fine grained controls for moderation.

Fixed

  • Patched a bug where leaving locale blank on a filter entry could cause all pages to throw an exception due to new locale handling behavior in mvc. The form now detects the implicit locale and treats it as missing until we can change locale handling everywhere.

Version 3.27.2

Released Thursday, February 6th, 2020

Fixed

  • Some TLD values could cause the edit button to redirect to a 400 page in tomcat due to the escapes present in the segment. This is both fixed in PrimeMVC and we use parameterized versions of the url to be safer

  • Fixed an error on the alerts queue where the dismiss and ignore buttons would display their java toString value instead of the HTML escaped value.

Version 3.27.1

Released Friday, January 24th, 2020

Enhancement

  • Improved HTML escaping to prevent XSS attacks

    • Freemarker now escapes all printed values unless explicitly told not to

  • Relaxed the requirements on ignorable characters to permit any character

    • We previously restricted this to a-z to prevent unknown side effects but found use cases for characters outside of that range. It is still recommended to consult support if you plan to change this setting as it could cause surprising and wide reaching changes in filter behavior.

  • Allow words to be repeated in the kids chat exact phrase builder

    • This has always been allowed via the API but the UI was being overly strict due to a missing flag to permit duplicates

Fixed

  • Fixed the verify tool not returning any matches if All Locales was selected. (Bug introduced in 3.27.0)

    • The All Locales previously mapped to null but a recent change in Prime MVC causes empty string (aka All Locales) to map to the Root locale. We now check for this locale.

  • Fixed an issue that would cause moderation pages to throw exceptions if Elasticsearch was being provided by AWS.

    • The status endpoint in AWS requires a leading slash and CleanSpeak was previously missing this slash. This was not required for standalone elasticsearch.

Version 3.27.0

Released Thursday, January 2nd, 2020

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • A new event has been added for Filter approvals. See Handling Events for additional details.

    • This event is fired when a change to the filter is approved. Filter approvals can be found in the UI by navigating to Filters Approvals.

    • The default transaction configuration for this event will require all webhooks to successfully acknowledge the event in order for the approval request to complete. You may optionally modify this default configuration by navigating to Settings System Webhooks.

    • To enable this new event for an existing Webhook, navigate to Settings Webhooks Events and enable the FilterApproval event.

    • Resolves GitHub Issue #12

  • Webhooks may now optionally select which events are to be sent. Navigate to Settings Webhooks Events to configure event selection.

  • Webhooks may now optionally define a free form description field to describe the intended purpose or details of the webhook.

  • Added a new webhook_manager role for managing webhooks. Users with the admin or webhook_manager role may manage Webhooks. See System Users to find the new role or use the User API to modify the users programatically.

Version 3.26.0

Released Friday, December 13th, 2019

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Content storage controls

    • When content storage is enabled, you may optionally configure CleanSpeak to store a percentage of the content to reduce the storage requirements of the application. While moderated content is always stored, using this configuration you may choose to only store a percentage of content that does not require moderation.

    • When content storage is enabled, you may also force all content that contains a match to be stored regardless of the previous setting.

    • Resolves GitHub Issue #7

Enhancement

  • Enhanced the URL filter accuracy

    • Explicitly fixes exampledotcom but also fixes many other embedded cases where dot is used instead of punctuation.

    • Resolves GitHub Issue #5

  • Improved the OpenID Connect workflow so that we catch more problems with the configuration

    • If the Token endpoint returns a 200 but the access_token field is missing from the JSON response then an error will be displayed indicating the problem is with the Token endpoint. Previously a generic error was displayed, indicating an issue with CleanSpeak.

    • If the Userinfo endpoint returns a 200 but does not have the email field in the JSON response then an error will be displayed indicating that the email field is required and your OpenID Connect configuration is incorrect.

Fixed

  • Fixed the TLD API to use name in the parameters/URI instead of using the one in the body on PUT. The previous behavior of supplying the name via the JSON body still works but is now deprecated.

Version 3.25.0

Released Monday, December 2nd, 2019

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Prometheus endpoint at /prometheus/metrics (On both the webservice and the management interface)

  • Top level domain UI and API.

    • Resolves GitHub Issue #4

    • Top level domains can now be modified directly in the UI or via the API so that you can further modify the URL filter behavior

    • Top level domains are used in CleanSpeak to identify URLs, this functionality already existed in CleanSpeak via a domains.properties file but has been transitioned to an API for ease of use.

  • URL whitelist entries can now be tagged for use with specific applications

    • Resolves GitHub Issue #6

    • By default any entries that are not tagged will be always used

    • Any entries that are tagged will only be used when an application that contains that same tag is supplied on the moderate endpoint

  • Whitelisted URL API

    • This API allows you to manage whitelisted URLs for the URL filter.

  • URL Whitelist Tag API

    • This API allows you to manage tags for the URL filter

Version 3.24.6

Released Tuesday, September 3rd, 2019

Enhancements

  • Upgraded Tomcat to 8.5.43

Fixed

  • The application cache would not preload the applications and would cause errors on the filter endpoint.

  • In rare conditions non blacklist entries would inherit the replace filter action.

    • The entry must be non blacklist entry

    • The match list must contain a blacklist entry match with mode replace

    • No match in the results can return anything higher than replace

    • The current match must be allow or author only with a user or content action enabled

Version 3.24.5

Released Wednesday, August 14th, 2019

Please Read

If you upgraded to this version please upgrade. This version has a flaw in the application cache that was fixed in 3.24.6.

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Enhancement

  • Application conditional reloading. Application cache will now only reload if a change has been made to reduce load on the database for situations where lots of applications may be desired. (Only affects webservice)

  • Application UI conditional paging. If you have more than 25 applications, paging controls will appear and restrict the number of applications visible at any given time. (Configurable)

Version 3.24.4

Released Thursday, August 8th, 2019

Fixed

  • Fixed sight engine integration. The JSON returned from the API has changed since the first implementation

Enhancement

  • Added partial nudity toggle to sight engine configuration in the System Settings. If enabled, the filter will be more strict about things like bikinis.

Version 3.24.3

Released Wednesday, August 7th, 2019

Enhancement

  • Silent mode startup. Added remaining required environment variables to allow silent mode to work. See Docker installation guide#Silent Configuration

    • Added LICENSE_ID and CLEANSPEAK_MEMORY environment variables for overriding cleanspeak.properties variables

Fixed

  • Removed revert button on blacklist whitelist entries pattern property (It was redundant since it was the only property)

  • Fixed the UI for blacklist whitelist entries not showing the correct state.

Version 3.24.1

Released Monday, July 22nd, 2019

Fixed

  • A missing translated message in the Moderation queue could cause a page not to render properly in some cases. If you encounter this symptom contact support and we can assist you with a work around.

  • When using a cleanspeak.properties file without the cleanspeak-app.search-servers property defined the value defined during the Search Engine configuration step of Maintenance Mode may not be persisted. The symptom may be noticed because the URL used to connect to the search engine will be defaulted to http://localhost:8021 and not the value displayed to you in maintenance mode. If you encounter this issue, it can be worked around by adding cleanspeak-app.search-servers= to the CleanSpeak configuration file. Once this value has been added the configuration step in the UI will work as expected.

Enhancement

  • The CleanSpeak Management Interface and CleanSpeak Webservice Linux processes will create a named directory in the /tmp filesystem during startup. This directory is expected to exist for the duration of the service process. In some environments this directory may deleted while the service process is still running. If this occurs and you attempt an action that requires this temporary directory to exist such as Import list or Generate Weekly Report an exception will occur. Additional safe guards have been put into place to protect against the possibility of this directory being removed during the process runtime.

Internal

  • Enhance processing of the cleanspeak.properties during initial parsing to ignored lines that begin with a # indicating they have been commented out.

  • Removed custom java.io.tmpdir environment variable in Linux init script. This change is related to the temporary directory enhancement described above.

Version 3.24.0

Released Monday, July 15th, 2019

Please Read

This migration will remove any existing external authentication configuration for Google or SAML identity providers. It is highly recommended to make a database backup in case you need to roll back. If you have previously been using Google or SAML identity providers for authentication to CleanSpeak please contact support so we can assist you with the migration. Moving forward only OpenID Connect will be supported as an external identity provider for CleanSpeak.

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • OpenID Connect authentication method. This feature allows for easy login to the majority of external identity providers. Contact support if you need assistance with this configuration. Navigate to Settings System Authentication to enable and configure OpenID Connect.

Removed

  • The following identity provider configurations have been removed from CleanSpeak. Contact support if you need assistance with migration from any of these legacy identity provider configurations.

    • Google Authentication. This configuration can be migrated to Google’s OpenID Connect, this has been tested and should provide you with a equivalent configuration.

    • SAML2 - If your current SAML v2 provider offers OpenID Connect this is the preferred migration. If your current SAML v2 provider does not offer OpenID Connect, FusionAuth can be used to bridge this gap. FusionAuth supports OpenID Connect and offers federation capability to SAML v2 Identity Providers.

    • Passport. This integration is now removed and deprecated. If you were previously using Inversoft Passport you may migrate to FusionAuth and utilize OpenID Connect.

Version 3.23.1

Released Monday July 1st, 2019

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Upgraded maintenance mode for a better workflow and better edge case handling

Fixed

  • Fixed edge case for whitelist

  • Updated jackson to fix CVE-2019-12086

  • Internal security improvements

Version 3.23.0

Released Wednesday May 15th, 2019

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Added a hard whitelist for the blacklist. Anything that matches a whitelist entry will never get matched by the blacklist.

  • Added the ability to bulk duplicate blacklist entries into another locale for per locale entry settings.

Fixed

  • Fixed replacement behavior when only meta matches exist

Version 3.22.2

Released Tuesday February 26th, 2019

Fixed

  • Tweaked an internal filter to prevent matches from being all digits if they contain replacements. If you want a match to contain only digits then it must have an explicit variation or filter entry with that exact text.

  • Calling an api with the wrong method now returns 405 instead of 501

Version 3.22.1

Released Friday December 28th, 2018

Fixed

  • Click event bindings were broken for some blacklist entry revert buttons. (Broken in 3.17.0 during the UI update)

Version 3.22.0

Released Tuesday October 23rd, 2018

Please Read

If you are using the Filter Content API, please be advised the new Unicode filter described below is enabled by default.

The default configuration may cause CleanSpeak to return matches you had not previously encountered. The reason for this is that if the message sent to CleanSpeak contains one or more characters from one of the default unicode match ranges the content will be rejected. A new match type will be returned of type unicode and the response will include the indices containing matched characters.

If you have questions or concerns about this new filter and how it may impact you please contact us at support@cleanspeak.com.

New

  • CleanSpeak now provides a Unicode filter designed to reject content containing characters in high order unicode ranges that are often used to bypass filters. For example many character glyphs exist in high order unicode that render as an alphabetic character but are not easily detected. CleanSpeak understands most of these replacement characters already but the possible variations are too numerous to account for every possible variation. The default configuration provided should cover most cases, but you may optionally modify or augment the unicode match ranges. Contact us at support@cleanspeak.com if you need assistance.

    • If you’re using the Filter Content API this filter will be enabled by default. Please review the advisory above.

    • If you’re using the Moderate Content API the default configuration will be to allow unicode matches, in other words CleanSpeak will not reject content based upon a unicode match. To enable this filter, find the Unicode filter rule in your Application configuration in the Other Filters tab, and find the Unicode filter options under the Advanced tab.

  • CleanSpeak will now ship with a set of ignorable characters that will allow the filter to treat certain characters similar to whitespace and punctuation. This feature is intended to allow more accurate detection of embedded words. The default configuration consists of the following three letters q, x and z.

    • Example, the word SxMxUxRxF is the word Smurf with x interleaved between each character. This word will now be identified as a match on smurf.

    • Example, the word xSmurf is the word Smurf prefixed with x. In some cases prior to this version, depending upon the filter mode, this word would not have been considered this a good match. Now this will will be identified as a match on smurf.

Fixed

  • In some cases Reverting a Blacklist Phrase in the Filter Approvals would cause an exception and would not be be reverted. This issue has been corrected. If you encounter this issue prior to version 3.22.0 contact support@cleanspeak.com for assistance.

Version 3.21.1

Released Saturday September 29th, 2018

Fixed

  • Oauth error handling bugs.

Version 3.21.0

Released Tuesday September 11th, 2018

New

  • Added Sightengine as an image filter provider.

  • Add back the variations column in the Blacklist filter listing in the UI.

Version 3.20.2

Released Monday August 20th, 2018

Fixed

  • When a complex input string containing many numbers is filtered, if your filter configuration has one to many distinguishable numbers an increased CPU load may occur which could cause performance degradation.

Version 3.20.1

Released Friday August 17th, 2018

Fixed

  • A CSS issue caused maintenance mode and the Filter test interface in the CleanSpeak webservice UI to render poorly. This was only a cosmetic issue and did not affect the CleanSpeak Management Interface. This issue was introduced in version 3.20.0.

Version 3.20.0

Released Wednesday August 15th, 2018

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Search Index Rebuild

A major version upgrade of Elasticsearch requires a full rebuild of the search index. This will be done automatically on startup, this may cause additional load on the system until completed. Content and user search will be incomplete until the process is complete.

New

  • Added sorting to user search and extended content search sorting options

  • Added query string searching to both content and user search

Fixed

  • When using PostgreSQL with a large amount of content stored in CleanSpeak, the performance may be degraded as moderates enter the queue.

  • Upgraded Elasticsearch to 6.3.1

  • URL whitelist entries were not approved after a Backup / Restore

  • Duplicate key violation when importing with new phrases

  • English was (almost) always selected for verify tool even if all locales was a better default.

  • Several display issues in the UI

Version 3.19.0

Released Thursday July 28th, 2018

New

  • Added HTML parsing to support html as a valid filter content type.

  • Added content type selector to Verify tool (Includes HTML, Video, Image, BBCode, and text)

  • Added CSV export option to the filter list export feature. The CSV option was removed in version 3.4.4 along with the CSV import feature.

Changed

  • The contentType parameter on the Filter Content API now defaults to text instead of detecting type. This may enhance performance when filtering large content.

Enhancement

  • Improved match highlighting accuracy when filter matches are displayed for content.

  • Add и (Cyrillic small letter i) as a possible replacement for the letter u. Example: fиck.

Fixed

  • Duplicate named sections in moderation queues for content types other than text.

  • Audit log reporting null for new dictionary entries.

  • Possible redirect loop during login when authenticating with an expired user, and then immediately authenticating with a different user who is not expired.

  • Edited content may not show the edited icon in the history column when viewed in the content search results.

  • When a very deep chain of users flagging other users, when the user is retrieved an exception may occur.

  • Upgraded JRE to 1.8.0+171

  • Upgraded Apache Tomcat to 8.5.31

Version 3.18.1

Released Tuesday June 5th, 2018

Fixed

  • Fixed bug that caused the content alert queue actions to fail

Version 3.18.0

Released Tuesday May 29th, 2018

Please Read

This migration erases any existing Blacklist Phrases created since version 3.15.0. If you wish to preserve this configuration, it is recommended to record any phrases you wish to keep before running this migration.

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Full support of regular expressions in Blacklist phrases. This feature supercedes the phrase feature introduced in version 3.15.0. For details see Blacklist Phrases. This feature overwrites the existing phrases entirely.

Enhancement

  • Additional API validation has been added to CRUD operations.

Fixed

  • Filter changes were not reloaded by the webservice after using the Restore API.

Version 3.17.2

Released Thursday May 10th, 2018

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Enhancement

  • More validation on the batch filter

  • Embedding behavior is improved when surrounded by numbers and punctuation

  • Embedding behavior is improved when finding the actual start of a blacklisted entry

  • Improved logging for license issues

Fixed

  • Updates caching (bug fixes)

Version 3.17.1

Released Thursday April 19th, 2018

Fixed

  • A potential issues exists when using the Batch Filter API or the Batch Moderate API in synchronous mode where intermittent Java Garbage Collection may impact filter performance.

Version 3.17.0

Released Tuesday April 17th, 2018

Please Read

CleanSpeak has received a major UI overhaul and you may pleasantly surprised when you login after the upgrade. Contact support@cleanspeak.com if you have further questions or would like to schedule a demo of the new UI.

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • A full UI refresh. We hope you like it.

Changed

  • To address some common confusion between the CleanSpeak Blacklist and Whitelist, the whitelist in the UI has been renamed to Kids Chat.

  • Notification Servers are now called Webhooks to better align with industry standard naming conventions.

Fixed

  • SMTP connection settings for Security type SSL will not function as expected.

  • A bug in the Dutch inflector may cause an exception in two letter verbs

  • Service startup scripts will tolerate Windows line returns when running on Linux or Mac OS X

Version 3.16.1

Released Saturday April 14th, 2018

Enhancement

  • Improved filter accuracy by handling embedded numbers and dictionary checks more effectively.

Version 3.16.0

Released Tuesday March 20th, 2018

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Video Filtering added using Amazon Rekognition. See how to setup video filtering for details.

Version 3.15.2

Released Monday January 29th, 2018

Fixed

  • The Common Terms report will now function properly from the Management Interface. This issue was introduced in Version 3.15.0

Internal

  • Reduced logging noise when running CleanSpeak on public networks

Version 3.15.0

Released Monday January 21st, 2018

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

Enhancement

  • Upgraded prime.js

Fixed

  • Minor bugs

Version 3.14.1

Released Monday December 11th, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Enhancement

  • Internal. Relax MySQL utf8mb4 enforcement to only character_set_server and collation_server. This should now pass validation on Google Cloud when character_set_server is configured to use utf8mb4.

  • Internal. Reduce log noise when webservices are on public networks. No external impacts.

  • Internal. Db schema null constraint corrected. No external impacts.

Fixed

  • JavaScript bug in Application Filter Rules may cause the username filter changes to not be saved.

Version 3.14.0

Released Friday November 24th, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Changed

  • API JSON response bodies now return empty objects. For example, the Filter API response will return an empty matches array even when no matches are returned.

  • Removed locale from verification cases as the interface allows you to choose/override this option anyways.

Enhancement

  • The Test a Notification Server feature provides better feedback when the notification server returns an unexpected status code.

  • The verify text feature has been redesigned to use the same display as the queue for better information on matches.

Fixed

  • The configured list of Whitelist Disallowed Phrases now properly displays a tag when it contains a space.

  • MaximumMatchLength on Email and URL filter wasn’t being honored when provided on the Filter API.

  • During filtering, blacklist entry ignores only considered a full token instead of just the original match text. This now checks both.

Version 3.13.2

Released Tuesday October 24th, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Fixed

  • Sending an integer or other non UUID string representation for the contentItems[x].id field on the Batch Moderate API returned 500 instead of a 400 with an error message.

  • Support sending an integer for the contentItems[x].id field on the Batch Moderate API.

  • Omitting the callbackHeaders caused the call to the callbackURL to fail on the Batch Moderate API.

  • Creating a Whitelist allowed entry without a locale was permitted on the Management Interface which will break the filter reload. This issue has been resolved and entries in this state will be corrected during upgrade. This issue was introduced in Version 3.11.0

Version 3.13.1

Released Thursday October 5th, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Integrated Image Filtering technology preview using Amazon Rekognition. Additional providers in plan for a future release.

Enhancement

  • The Content Delete service has been enhanced to handle large data sets without adding excessive database load.

Fixed

  • Minor UI Application configuration bugs.

Version 3.13.0-RC.1

Released Thursday September 7th, 2017

Enhancement

  • The filter now returns at least one match from every root word found. This means that all fully contained matches with separate entries will now be returned. (Both "you suck", and "suck" could be returned instead of just "you suck")

Fixed

  • When using CleanSpeak Moderation queues on a secured connection, YouTube URLs starting with http:// may not display correctly when using the embedded video player.

Version 3.12.1

Released Wednesday August 30th, 2017

Enhancement

  • Setup wizard no longer prompts the user for License Id when coming from their CleanSpeak account page.

  • Better embedded video player support for vimeo.com, ted.com videos in the moderation queue.

Version 3.12.0

Released Wednesday August 23rd, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Added the ability to make dictionary entries conditional and behave like a whitelist.

  • Filter search now includes dictionary tags.

  • Applications now can have dictionary tags to modify filter behavior (API/Management Interface)

Fixed

  • Whitelist allowed entry sorting on tags.

Version 3.11.0

Released Wednesday June 21st, 2017

New

Version 3.10.0

Released Wednesday June 7th, 2017

New

Fixed

  • Added missing APIs to the API key endpoint configuration.

Version 3.9.2

Released Wednesday May 31st, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Added CORS filter for webservice API requests.

Fixed

  • Added additional validation to Whitelist and Blacklist APIs.

Version 3.9.1

Released Friday May 27th, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

Version 3.8.1

Released Wednesday May 3rd, 2017

Enhancement

  • Added totalResults to blacklist search api results.

Fixed

  • Blacklist APIs now only return entries that are active in the filter.

Version 3.8.0

Released Tuesday May 1st, 2017

New

Version 3.7.7

Released Thursday March 30th, 2017

Enhancement

  • Reduce false positives when partial matches are embedded inside of a UUID.

Version 3.7.6

Released Tuesday March 21st, 2017

Enhancement

  • Better handling of numbers in between dictionary words.

Version 3.7.5

Released Thursday March 9th, 2017

Fixed

  • A filter bug may cause specific variations with trailing duplicate characters in mixed case to be ignored by the filter.

  • When emailing moderators on escalation events in some scenarios a user with access to All Applications may not be identified to receive the event notification.

  • Better differentiate between symbols and word characters when detecting replacement characters in a match.

Version 3.7.4

Released Monday February 13th, 2017

Fixed

  • An error may keep filter rules from being configured in the Management Interface for an application. This issue was introduced in version 3.7.0.

Version 3.7.3

Released Monday February 13th, 2017

Fixed

  • An error may cause the Weekly Report from being sent for the first time.

Version 3.7.2

Released Thursday February 9th, 2017

Fixed

  • An error may occur when filtering content using the Moderate Content API with Whitelist Filter Rules with a configured alert.

Version 3.7.0

Released Monday February 6th, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Added transaction support for content editing, content deleting, user actions and notification servers. You can now manage whether or not notification servers need to successfully respond to notifications for various events.

  • Added a notification server testing system for developers to quickly test notification servers.

Fixed

  • Multi-lingual handling for various words like ass and dass

Version 3.6.2

Released Monday January 25th, 2017

Fixed

  • Bug in URL filter that may cause a 500 response code. This issue was introduced in 3.5.0.

Version 3.6.1

Released Monday January 23th, 2017

Fixed

  • Email not sent when escalating item from approval queue when enabled using feature added in 3.6.0.

Version 3.6.0

Released Friday January 20th, 2017

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Show changes for list approvals and allow modification prior to approval.

  • Batch filtering API.

  • Weekly PDF activity reports.

  • Configuration to allow moderators to be notified when content is escalated.

Enhancement

  • Upgraded from Apache Tomcat 8.0.12 to 8.5.9.

  • The version is no longer used in the path name inside of the installation packages for Apache Tomcat and ElasticSearch.

Fixed

  • Some JavaScript errors may prevent a moderator from editing content in a moderation queue. This issue was introduced in 3.5.1.

Version 3.5.1

Released Thursday December 15th, 2016

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Initial support for mobile devices using a responsive design. Navigation is provided as a left hand navigation menu.

Enhancement

  • Upgrade prime.js. Now using native .bind() support for maintaining the correct this reference.

  • Encrypt Saved Request cookies.

Fixed

  • Accessing the Status action from the Management Interface fails to show the status of the webservice. System → Status

  • Improve embedding validation for matches containing replacement characters. Example: [0cat0] is

  • Bug in URL filter that may cause a 500 response code. This issue was introduced in 3.5.0.

Version 3.5.0

Released Tuesday October 25th, 2016

New

  • CleanSpeak no longer ships with a default account. A setup wizard is now provided for initial configuration.

Enhancement

  • Add 4 as a possible replacement character for a.

Fixed

  • The URL filter may miss a URL if the string begins with a dot separator.

Version 3.4.11

Released Thursday September 26th, 2016

Changed

  • Deprecate checkOutUser field in the Retrieve a User API and add checkOutUserId.

Version 3.4.10

Released Thursday September 15th, 2016

Enhancement

  • Support for CSV file type of application/vnd.ms-excel in the batch processing filter. Appropriate messaging will be displayed when an unsupported file type is attempted to be uploaded.

Version 3.4.8

Released Wednesday September 14th, 2016

Enhancement

  • Enhancement. Increase maximum file size allowed for upload for batch CSV filtering from 50 MB to 100 MB.

Fixed

  • Missing a few Cyrillic characters for Latin replacement characters. This allows some matches to get past the filter if Cyrillic characters are inter-mixed with Latin.

Version 3.4.7

Released Tuesday September 6th, 2016

Enhancement

  • Internal License validation changes

Version 3.4.6

Released Tuesday September 6th, 2016

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Fixed

  • When using the verify text tool, username matches will also have a link to the matched root entry.

  • Better handling of numbers and extra characters before or after words.

  • Some blacklist entries had a variation matching the root entry which limit the ability to modify the entry. The blacklist entries have been fixed in the shipped lists, and the entry validation has been fixed to resolve the issue at runtime.

Version 3.4.5

Released Thursday August 18th, 2016

Fixed

  • When using the /content/item/filter API and enabling the username filter, no other match types are returned. This bug was introduced in version 3.4.0.

Version 3.4.4

Released Tuesday August 2, 2016

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • CSV Filtering

  • Backup / Restore API

Version 3.4.3

Released Friday July 1st, 2016

Fixed

  • Import of the Arabic language using the import stock list feature was failing due to duplicate variations. Prior to this version the Arabic language can be imported using the Import feature.

Version 3.4.2

Released Tuesday June 28th, 2016

Fixed

  • Import stock lists was not loading variations or ignore cases.

Version 3.4.1

Released Wednesday June 27th, 2016

Enhancement

  • Show locale for filter matches in the filter match table displayed in the content queues.

Fixed

  • systems and sys-v init issues.

Version 3.4.0

Released Wednesday June 22nd, 2016

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Email configuration has moved from cleanspeak.properties file to the Management Interface. Your existing configuration has been migrated and should continue to operate. See Email Configuration.

New

  • Username Filter for use on the Filter or Moderate request

  • Whitelist Builder

  • Automated Actions per application

  • Whitelist Filter Rules for Application

Enhancement

  • Filter Export provides a selector for languages.

  • Email Template and BBCode editor now uses CodeMirror text editor to provide syntax highlighting and some basic formatting.

Fixed

  • Import stock filter lists on Windows may fail.

  • Using maintenance mode to bootstrap CleanSpeak database configuration on PostgreSQL may fail.

Version 3.3.1

Released Tuesday March 15th, 2016

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Fixed

  • Integration with Passport.

Version 3.3.0

Released Friday March 11th, 2016

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

The authentication token previously configured in the cleanspeak.properties file is no longer optional. All REST API requests require authentication. The API keys are now configured through the Management Interface, see API Authentication for more information.

New

  • New Content Analysis Reports

  • Installation and database configuration can now be performed through a maintenance mode

  • Enhancements to the image queue

  • Email Templates

  • Authentication configuration has moved from cleanspeak.properties to being managed through the Management Interface

Version 3.2.0

Released Saturday February 6th, 2016

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • New Approval Image Queue to improve the efficiency of image moderation. Enable this feature by marking your application as Image Only.

Version 3.1.8

Released Tuesday February 2nd, 2016

Enhancement

  • Show the return status code to a user when a request to the configured notification server has failed.

Fixed

  • When running CleanSpeak with a context path defined, some images may not display correctly.

  • In 3.1.7 we build links to black list matches using a unique identifier. When viewing content in the queue that was filtered prior to 3.1.7 the links will not function properly.

  • Some black list filter matches were missed when containing repeating mixed case characters.

Version 3.1.7

Released Monday January 18th, 2016

Enhancement

  • Black list match links when viewed in the moderation queue or the verify text tool will now take the user directly to the black list entry instead of using a text search.

  • Support Gmail SMTP server with TLS

Version 3.1.6

Released Wednesday December 10, 2015

Enhancement

  • Internal changes only.

Version 3.1.5

Released Wednesday December 10, 2015

Fixed

  • User and Location (threaded-view) links were broken in the content view

  • Missing Windows Service .exe for CleanSpeak Search Engine

  • Filter only users could see the "Approvals" menu item

  • Database Languages ZIP file wasn’t properly being released to the website

Version 3.1.4

Released Wednesday December 9, 2015

Fixed

  • An unexpected render of poorly formed BBCode may cause a user to be stuck in the alert queue. This has been fixed to ensure regardless of the BBCode rendering in the queue, the user can always be dismissed properly.

  • Enhanced our ability to find repeating characters when separated by punctuation.

  • Sending null values in JSON to the filter endpoint was returning a 500 status code. This behavior has been corrected.

Enhancement

  • Viewing filter matches in content search or the moderation queue now allows you to view all matches in a content part in a table format.

  • CleanSpeak web services can now be run as Windows services.

Version 3.1.3

Released Monday December 7, 2015

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Version 3.1.2

Released Thursday October 8, 2015

Enhancement

  • Russian: Updated black list entry: лох . Changed filter mode from exact_match to non-embeddable. This allows 'о' to be replaceable with english 'o'.

  • Russian: Updated black list entry for бля. Changed filter mode from exact_match to filter mode to embeddable and added an ignore case for бляха.

Fixed

  • Fixed: URL filter has been enhanced to more accurately find the end of a top level domain.

  • Fixed: Enhanced our ability to identify YouTube video URLs in the moderation queue to build an inline player. Add proper sizing to the HTML 5 video tag. Use the Google iFrame API to support the HTML5 YouTube player while still falling back to the Flash Player when the browser does not support the HTML 5 video player.

Version 3.1.1

Released Monday October 5, 2015

Fixed

  • OAuth and SAML login errors

Version 3.1.0

Released Saturday September 26, 2015

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

New

  • Added a new moderation queue called the Content Alert Queue.

    • You can put content flags into this queue or put content directly into this queue using a parameter on the request.

    • This queue allows the content to be reviewed and dismissed.

    • You can also optionally add content actions, which will allow you to take actions on content in the Content Alert Queue

Version 3.0.14

Released Thursday September 24th, 2015

Fixed

  • Fixed: A condition was identified where an attempt to approve a deletion of a white list entry may fail and an error would be displayed in the CleanSpeak Management Interface stating An unexpected error occurred.

Version 3.0.13

Released Monday September 21st, 2015

New

  • Added parameters to reduce the filter results on the /content/item/filter endpoint

Version 3.0.12

Released Friday September 11th, 2015

New

  • Added User deletion system that allows all information and content for multiple users to be completely deleted from CleanSpeak (great for COPPA compliance)

Version 3.0.11

Released Tuesday September 8th, 2015

New

  • Added the ability to disable url, email and phone number filters on a per request basis using the /content/item/filter endpoint. See /content/item/filter documentation.

Version 3.0.10

Released Wednesday September 2nd, 2015

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Fixed

  • Fixed possible error in filter approvals that can cause blacklist entries to lose all their tags

Version 3.0.9

Released Wednesday July 15th, 2015

Fixed

  • When viewing BBCode content in the moderation queue it is possible that a filter match found embedded within a BBCode attribute could cause the HTML to be rendered invalid.

Version 3.0.8

Released Thursday July 9th, 2015

Fixed

  • A customer identified edge case scenario where a conjugated black list match would ultimately not be returned in the filter results.

Version 3.0.7

Released Monday June 8th, 2015

Enhancement

  • Allow a user to be flagged when they currently have content in the approval queue. Previously when the /content/item/flag API is used and the content is currently in the approval queue, a 409 status code is returned. Some customers have a workflow that allows for content to be visible and thus flagged by another user while the content is still in the approval queue. This limitation is now removed, and a 200 will be returned instead.

Fixed

  • A filter match may not be returned if the word also exists in the dictionary, was sent in as upper case or mixed case, and was followed by a number.

  • A filter match with trailing characters may not be returned when the match is in mixed or upper case.

Version 3.0.6

Released Tuesday May 12th, 2015

Enhancement

  • Improved filtering of embedded numbers in profanity matches. For example, he222222ll will now match hell since the numbers are now treated similarly to punctuation.

Version 3.0.5

Released Tuesday May 5th, 2015

Database migration

The database schema has changed and an upgrade is required for this version of CleanSpeak. You will be prompted to upgrade the database by maintenance mode before you may login.

See Upgrades/Patches for more information about database migrations.

Fixed

  • /content/item/moderate endpoint was returning a replacement string with allowed, but alerted, matches were "starred" out when the contentAction is replace. Now, the allowed, or authorOnly, filter matches will not be replaced with the replacement character.

Version 3.0.3

Released Wednesday April 8th, 2015

Enhancement

  • Improved compatibility end-point to accept invalid UUIDs for senderUID and receiverUID (defaults invalid values to all 0s)

Fixed

  • Fixed top user report to sort the results properly

Version 3.0.2

Released Thursday March 5th, 2015

Fixed

  • BBCode fixes to improve rendering in queues and content search

Version 3.0.1

Released Friday February 27th, 2015

Fixed

  • Bug fixes for the 2.3 compatibility API end-point

Version 3.0.0

Released Tuesday February 24th, 2015

Enhancement

  • SAML improvements for role and application handling

  • Locale improvements in the Management Interface

  • Fixes to the Management Interface and WebService for filter-only customers. You no longer need to run the Search Engine to start CleanSpeak.

  • Minor Management Interface improvements for filter-only customers