Move up key phrase filter to improve results

This will prevent false-positives for the following scenario:

1. There are multiple matches, with a smalller match that id contained
2. It is filtered because of containment in the larger match
3. The larger is discarded because of not having keyphrases
4. There is no match left, we now have a false negative

As a practical example:

1. In query there is `licensed under the AGPL 3.0`
2. In rule1 I have `AGPL 3.0`
3. In rule2 I have `licensed under the {{AGPL 3.0 license}}`
3. The query will initially match both rules
4. The match to rule1 may be discarded as contained in the rule2 match
5. The match to rule2 may be discarded as missing a key phrase

By moving the key phrase filter up before the containment filter the
larger match may be filtered out, giving the smaller matches a chance
to stay.

Signed-off-by: Mike Rombout <mike.rombout@elastisys.com>

Author: mrombout

src/licensedcode/data/rules/apache-2.0_1026.RULE

@@ -0,0 +1 @@
+This SDK is distributed under the Apache License, Version 2.0, see LICENSE for more information.

src/licensedcode/data/rules/apache-2.0_1026.yml

@@ -0,0 +1,3 @@
+license_expression: apache-2.0
+is_license_notice: yes
+relevance: 100

src/licensedcode/data/rules/apache-2.0_571.RULE

@@ -1,6 +1,6 @@
 <licenses>
         <license>
-            <name>Apache License 2.0</name>
+            <name>{{Apache License 2.0}}</name>
             <url>http://repository.jboss.org/licenses/apache-2.0.txt</url>
             <distribution>repo</distribution>
         </license>

src/licensedcode/data/rules/apache-2.0_url_15.RULE

@@ -1 +1 @@
-  <licenses>    <license>      <name>Apache License, Version 2.0</name>      <url>http://apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>
+  <licenses>    <license>      <name>{{Apache License}}, Version 2.0</name>      <url>http://apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>

src/licensedcode/data/rules/bsd-new_531.RULE

@@ -1,6 +1,6 @@
 <licenses>
     <license>
-      <name>Berkeley Software Distribution (BSD) License</name>
+      <name>{{Berkeley Software Distribution}} (BSD) License</name>
       <url>http://www.opensource.org/licenses/bsd-license.html</url>
       <distribution>repo</distribution>
     </license>

src/licensedcode/data/rules/mpl-1.0_14.RULE

@@ -1,6 +1,6 @@
 MPL:
 
-"The contents of this file are subject to the Mozilla Public License Version 1.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/
+"The contents of this file are subject to the {{Mozilla Public License}} Version 1.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/
 
 Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.
 

src/licensedcode/data/rules/mpl-1.0_8.RULE

@@ -1,4 +1,4 @@
- * The contents of this file are subject to the Mozilla Public License
+ * The contents of this file are subject to the {{Mozilla Public License}}
  * Version 1.0 (the "License"); you may not use this file except in
  * compliance with the License. You may obtain a copy of the License at
  * http://www.mozilla.org/MPL/

src/licensedcode/data/rules/mpl-1.1_36.RULE

@@ -1,6 +1,6 @@
 The MPL v1.1:
 
-   The contents of this file are subject to the Mozilla Public License
+   The contents of this file are subject to the {{Mozilla Public License}}
    Version 1.1 (the "License"); you may not use this file except in   
    compliance with the License. You may obtain a copy of the License at
    http://www..com/mpl.html

src/licensedcode/data/rules/mpl-1.1_40.RULE

@@ -1,4 +1,4 @@
-The contents of this file are subject to the Mozilla Public License Version 1.1
+The contents of this file are subject to the {{Mozilla Public License}} Version 1.1
 (the License); you may not use this file except in compliance with the License.
 You may obtain a copy of the License at http://www.mozilla.org/MPL/
 Software distributed under the License is distributed on an AS IS basis,

src/licensedcode/data/rules/npl-1.1_1.RULE

@@ -1,5 +1,5 @@
-* The contents of this file are subject to the Netscape Public
- * License Version 1.1 (the "License"); you may not use this file
+* The contents of this file are subject to the {{Netscape Public
+ * License}} Version 1.1 (the "License"); you may not use this file
  * except in compliance with the License. You may obtain a copy of
  * the License at http://www.mozilla.org/NPL/
  *

src/licensedcode/data/rules/proprietary_133.RULE

@@ -1,4 +1,4 @@
- * The contents of this file are subject to the KnowledgeTree Public
+ * The contents of this file are subject to the {{KnowledgeTree}} Public
  * License Version 1.1 ("License"); You may not use this file except in
  * compliance with the License. You may obtain a copy of the License at
  * http://www.ktdms.com/KPL

src/licensedcode/match.py

@@ -1498,6 +1498,10 @@ def _log(_matches, _discarded, msg):
     all_discarded.extend(discarded)
     _log(matches, discarded, 'GOOD')
 
+    matches, discarded = filter_key_phrase_spans(matches)
+    all_discarded.extend(discarded)
+    _log(matches, discarded, 'KEY PHRASES')
+
     matches = merge_matches(matches)
     if TRACE: logger_debug(' #####refine_matches: before FILTER matches#', len(matches))
     if TRACE_REFINE:
@@ -1540,10 +1544,6 @@ def _log(_matches, _discarded, msg):
         all_discarded.extend(discarded)
         _log(matches, discarded, 'HIGH ENOUGH SCORE')
 
-    matches, discarded = filter_key_phrase_spans(matches)
-    all_discarded.extend(discarded)
-    _log(matches, discarded, 'KEY PHRASES')
-
     if merge:
         matches = merge_matches(matches)
 

tests/licensedcode/data/datadriven/lic1/erlware-relx.txt.yml

@@ -1,3 +1,4 @@
 license_expressions:
+  - apache-2.0
   - apache-2.0
   - apache-2.0

tests/licensedcode/data/datadriven/lic1/jenkinsci-role-strategy-plugin.txt.yml

@@ -1,2 +1,3 @@
 license_expressions:
   - mit
+  - mit

tests/licensedcode/data/datadriven/lic1/uwiger-parse_trans.txt.yml

@@ -1,2 +1,3 @@
 license_expressions:
+  - erlangpl-1.1
   - erlangpl-1.1