Clover13

I got it, thanks @Ryan Ashbrook

Whitelisted the server IP at:  Security → WAF → Tools → IP Access Rules

43 minutes ago, Ryan Ashbrook said:

Most likely - on their site, the Pro account advertises that they have blocking / challenging of automated traffic. In most cases, this is simply detecting that the request is coming from an IP from a data center and not an actual ISP (I should know, our corporate VPN gets hit by this all. the. time.).

What fix did you implement for the corp VPN? 🙂

7 minutes ago, Ryan Ashbrook said:

It's likely that Cloudflare is detecting that the IP the request is coming from is from a server / data center, and is blocking it as automated traffic.

When the Admin CP tests for these things, it makes an HTTP request to https://example.com/login/ for FURLs, and to https://example.com/api/core/hello for the API.

But only the CF Pro account and domain would do this, not the 2 other CF Free accounts and domains?

Figured I'd ask in this forum to see if any other site owners or developers have any insight into this issue I'm experiencing on one of my sites.

The setup:

• 1 server that hosts 3 IPS sites.  All sites running the same version of IPS (latest patched).
• 1 site uses Cloudflare Pro, 2 sites use Cloudflare Free
• CF Pro site gets a warning/error in the AdminCP indicating rewriting does not seem to be working, however the FURLs do work OK.
• CF Pro site does not work with the AdminCP API.  It loops to the same screen with Continue despite the .htaccess being in the proper /api directory.
• The other 2 sites using CF Free do not experience this issue.
• Disabling CF on the CF Pro site resolves the issues with the errors/warnings and the API screen works fine (without changing anything else).  This isolates the problem to something with CF Pro however I've been through every setting and compared to the other 2 CF Free sites and cannot identify anything that would be causing this.

Anyone have any ideas?

2 hours ago, Jim M said:

The software is getting an unexpected response from your server so is showing the .htaccess information as it is acting like it isn't present. The items mentioned were just suggestions, there may be something else entirely and it will take someone familiar with your hosting setup to fully look into that. Do you use CloudFlare or another firewall product? Is it stopping this from running functionally? Is there a rouge Apache setting only for this site? Tons of things to investigate.

Alright, so you mentioned CF so I disabled that and low and behold it worked again.  Re-enabled and after a short while, it broke again.  That lead me to think it was a corrupt cache but purging the entire cache did not resolve it.  The one primary difference is this one site is on a CF Pro account and the others are not.  So now it's matter of what on CF changed with Pro accounts recently that influences this.

Also nothing identified in the AdminCP Support panel.  Running latest version and 0 critical issues and 0 recommendations.

17 hours ago, Jim M said:

You would want to ensure your configuration for all sites are the same. Just because they are all on the same server, does not mean they are configured the same. Maybe there's a php.ini or .htaccess causing something screwy to happen or the opposite, the php.ini or .htaccess is required to undo something in the core configuration. Your host would need to assist you with this if you are unsure how to check these.

I reviewed the php.ini and installed php packages and versions.  Both the working and non-working sites are identical.

Likewise, with the .htaccess at both the root and /api levels was compared and tested.  So I'm at a dead end here as to where to check.  Is there not any log anywhere that would identify what is causing the error or causing the API admin page to go into an infinite loop of Continue despite the .htaccess file (downloaded from that same screen) being present in the /api directory?

3 hours ago, Marc Stridgen said:

The code for each is exactly the same, so really it could only be hosting level I would think

Heh, and the host says their accounts for each site are identical (ala cPanel, etc), so it can only be the software.  I wonder if @Thomas Hop figured it out.

Any chance it could be some permissions issue? Where else can I look to try and narrow down on this problem?  Any system/error logs that would identify what isn't working and causing that error?

6 minutes ago, Nathan Explosion said:

Just because you didn't specifically menion it - after putting the file in the /api folder, did you rename it so that it comes into effect?

The file downloaded is called htaccess, so did you rename it to .htaccess when you put it in the /api folder?

Yes it was renamed correctly.  There was also an existing one in there that worked for a long time until recently, which I subsequently renamed (.htaccess-20240415) and reuploaded a IPS ACP generated .htaccess file.  Same result.  Just an infinite "Continue" loop at the API menu of ACP.

8 hours ago, Marc Stridgen said:

Check you havent got anything else rewriting below that level which is not IPS related that may be interfering with it on those 2 sites.

I did, I removed it all down to just the IPS rules and it still resulted in the same issue.  Also worth adding, the htaccess file that was working wasn't changed for years.  So something changed, either at the host level or at the IPS level.  I can't tell which because it's a single server with 3 instances of IPS runnings where two work and one doesn't.  The FURL rewrites seem to work fine, but the API menu in the ACP keeps prompting me to install the htaccess file in the api directory (which I did) but hitting Continue does nothing but reload the initial page.  

Something to add that is important, my rewrites on the "broken" site do seem to work as FURLs work.  However the REST API will not, it keeps asking me to upload the htaccess file, which is there and has been there and worked before.  I reuploaded it with the same result of not working.  Likewise that error shows on the Rewrite URLs config setting in the Admin CP.

@Thomas Hop did you ever figure out the fix for this?

I'm having a similar problem.  Threes sites running on the same server and one of them has this issue while the other two don't.  All running the same version of IC (latest patched).  Having host review it, but am baffled at what could be different for one site vs the two others when they're running on the same server with what should be the same setup.  

FWIW I didn't need to store revisions for my use case, so I turned that off and it will work as it should.  If you need revisions you'll have to wait for a fix or do something custom as @teraßyte mentioned.

Possible bug.  The POST does work for most of the input I'm passing, but failing under certain cases.  Would prefer to PM the details as it's a specific case that is not working and I'm not really sure what is causing it, so best to provide the actual REST POST data and the subsequent query being created by the IPS software that is failing (as seen in the System Logs).  Will also provide a working case for comparison.

I'm on like my 6th review now trying to figure out what they need or want to verify the app and business.  I upload exactly what they require, I send messages asking for feedback, I try to send messages to their Support Chat and receive no answers.  It's been denied twice and the supporting reasoning was either corrected or incorrect (as that exact information was indeed provided).  The other times, the review was just cancelled with no reason (after weeks+ of waiting).

6 minutes ago, teraßyte said:

@Clover13 To reproduce the bug, the database must have the Store revisions option enabled. Most likely the test was made on a database with it disabled.

I initially thought too the error was coming from saving the record to the database, only after re-checking your last screenshot I noticed it was a revision instead. 😅

I see how you tracked it down in the PHP code, I'll try to do that next time as well.  I'm not too PHP savvy, but I see the stacktrace there and how you tracked it down, thank you for doing that! 👍

Also confirming with revisions OFF for the Pages DB, it does work and returns a 201 (CREATED)

2024-03-21 11:12:21 - INFO - post_cms_record_url: http://localhost/api/index.php?/cms/records/1
2024-03-21 11:12:21 - INFO - post_data: {'category': 1, 'author': 1, 'fields[1]': 'Title 123', 'fields[2]': 'Content ABC'}
2024-03-21 11:12:21 - DEBUG - Starting new HTTP connection (1): localhost:80
2024-03-21 11:12:21 - DEBUG - http://localhost:80 "POST /api/index.php?/cms/records/1&key=ef2f4f1687b901d3c962f074dcd4528e HTTP/1.1" 201 1109

4 minutes ago, teraßyte said:

Looking again at the screenshot, the error is being thrown when a revision for the record is added to the database, not when the record itself is added. If you're adding a new record, it shouldn't store a revision. A revision should be saved only when you edit a record.

 

The problem is in /applications/cms/api/records.php in the _createOrUpdate() function (lines 424-443):

			/* Store a revision before we change any values */
			if ( $item::database()->revisions )
			{
				$revision = new \IPS\cms\Records\Revisions;
				$revision->database_id = $item::$customDatabaseId;
				$revision->record_id   = $item->_id;
				$revision->data        = $item->fieldValues( TRUE );
				
				if ( $this->member )
				{
					$memberId = $this->member->member_id;
				}
				else
				{
					$memberId = $item->author()->member_id;
				}
				
				$revision->member_id = $memberId;
				$revision->save();
			}

 

The IF check should also check if you're editing a record because when adding a new one there is no record ID available yet (thus the column NULL error):

if ( $type == 'edit' AND $item::database()->revisions )

Thanks @teraßyte.  I noticed the edit aspect and had mentioned that earlier.  I would think any POST to create a new record would fail in this case (which is what I'm experiencing across 3 different Pages DBs), but @Marc Stridgen wasn't able to reproduce it.  @Marc Stridgen can you confirm you tested on 4.7.16?

4 hours ago, Marc Stridgen said:

In the first instance, try removing the 2 hooks you have in play there, and see if that resolves the problem. Im not able to replicate it on this end, and since you have this on local, its not too easy to debug

I'll do some more testing today.

FWIW, the GET to the hello endpoint works fine.

I'll also get appropriate errors when the POST executes and determines something like an invalid category or author.

2024-03-21 10:21:24 - INFO - post_cms_record_url: http://localhost/api/index.php?/cms/records/1
2024-03-21 10:21:24 - INFO - post_data: {'category': 99, 'author': 1, 'fields[1]': 'Title 123', 'fields[2]': 'Content ABC'}
2024-03-21 10:21:24 - DEBUG - Starting new HTTP connection (1): localhost:80
2024-03-21 10:21:24 - DEBUG - http://localhost:80 "POST /api/index.php?/cms/records/1&key=ef2f4f1687b901d3c962f074dcd4528e HTTP/1.1" 400 72

Ran 1 test in 0.094s

OK
Error: 400
Response: {'errorCode': '1T306/5', 'errorMessage': 'NO_CATEGORY'}

Regarding the hooks being disabled, that did not resolve the issue.

2024-03-21 10:25:57 - INFO - post_cms_record_url: http://localhost/api/index.php?/cms/records/1
2024-03-21 10:25:57 - INFO - post_data: {'category': 1, 'author': 1, 'fields[1]': 'Title 123', 'fields[2]': 'Content ABC'}
2024-03-21 10:25:57 - DEBUG - Starting new HTTP connection (1): localhost:80
Error: 500
Response: {'errorCode': 'EX1048', 'errorMessage': 'UNKNOWN_ERROR'}
2024-03-21 10:25:58 - DEBUG - http://localhost:80 "POST /api/index.php?/cms/records/1&key=ef2f4f1687b901d3c962f074dcd4528e HTTP/1.1" 500 72
 

FWIW...

# With API Key in request params returns 500 and EX1048
response = requests.post(post_cms_record_url, data=post_data, params=request_params)

# With API Key base64 encoded in headers
# Error: 401
# Response: {'errorCode': '3S290/7', 'errorMessage': 'INVALID_API_KEY'}
response = requests.post(post_cms_record_url, data=post_data, headers=headers)

# With API Key in auth returns 500 and EX1048
response = requests.post(post_cms_record_url, data=post_data, auth=auth)

24 minutes ago, Adriano Faria said:

EX1048 means that a column cannot be NULL.

Right and I believe that's due to the first screenshot I posted where it's expecting a record_id when one doesn't exist yet since this is creating a record, not editing one.  Unless that DB update is made after the record is created (but not committed yet), I don't know the sequencing or transactionalization of their DB commands for the API.

Maybe it's due to the api key being a request param instead of a header, but the header approach wouldn't work for me before this current version of the software.

8 minutes ago, teraßyte said:

500 error code aside, the API should also return a separate code/message about the issue. Have you tried checking that?

2T306/4	INVALID_DATABASE
1T306/5	NO_CATEGORY
1T306/6	NO_AUTHOR
2T306/G	NO_PERMISSION
1T306/D	TITLE_CONTENT_REQUIRED
1S306/E	UPLOAD_FIELD_NOT_OBJECT
1T306/F	UPLOAD_FIELD_NO_FILES
1T306/G	UPLOAD_FIELD_MULTIPLE_NOT_ALLOWED
1T306/H	UPLOAD_FIELD_IMAGE_NOT_SUPPORTED
1T306/I	UPLOAD_FIELD_IMAGES_ONLY
1T306/J	UPLOAD_FIELD_EXTENSION_NOT_ALLOWED

I dropped the data parsing for the non-200 case, here is that output:

2024-03-20 16:38:05 - INFO - post_cms_record_url: http://localhost/api/index.php?/cms/records/1
2024-03-20 16:38:05 - INFO - post_data: {'category': 1, 'author': 1, 'fields[1]': 'Title 123', 'fields[2]': 'Content ABC'}
2024-03-20 16:38:05 - DEBUG - Starting new HTTP connection (1): localhost:80
2024-03-20 16:38:05 - DEBUG - http://localhost:80 "POST /api/index.php?/cms/records/1&key=ef2f4f1687b901d3c962f074dcd4528e HTTP/1.1" 500 72

Ran 1 test in 0.179s

OK
Error: 500
Response: {'errorCode': 'EX1048', 'errorMessage': 'UNKNOWN_ERROR'}

It's just a python script, I put this together to post to the default Articles Pages DB and get a 500 as well.

 def test_api_cms_post(self):

        post_cms_record_url = f"{self.API_URL}{self.CMS_RECORDS_ENDPOINT}1"
        logging.info(f"post_cms_record_url: {post_cms_record_url}")

        # Request parameters with API key included in the URL
        request_params = {
            'key': self.API_KEY
        }

        post_data = {'category': 1, 'author': 1, 'fields[1]': 'Title 123', 'fields[2]': 'Content ABC'}
        logging.info(f"post_data: {post_data}")

        # Make a POST request to the API endpoint
        response = requests.post(post_cms_record_url,  data=post_data, params=request_params)

        # Check if the request was successful (status code 200)
        if response.status_code == 200:
            # Parse the JSON response
            data = response.json()

            # Process the data as needed
            print("Response:", data)
        else:
            # Print an error message if the request was not successful
            print(f"Error: {response.status_code}")

Output:

2024-03-20 16:13:43 - INFO - post_cms_record_url: http://localhost/api/index.php?/cms/records/1
2024-03-20 16:13:43 - INFO - post_data: {'category': 1, 'author': 1, 'fields[1]': 'Title 123', 'fields[2]': 'Content ABC'}
2024-03-20 16:13:43 - DEBUG - Starting new HTTP connection (1): localhost:80
Error: 500
2024-03-20 16:13:43 - DEBUG - http://localhost:80 "POST /api/index.php?/cms/records/1&key=ef2f4f1687b901d3c962f074dcd4528e HTTP/1.1" 500 72

Ran 1 test in 0.140s

OK

48 minutes ago, Jim M said:

Are you getting this with any post made to this endpoint or specifically with a certain request?

I tried another Pages DB and POST call that worked before the update and it too is getting the same error.

Was working on something new and started issuing POSTs to a Pages DB but it's erroring with a 500.  System log entry below.

Being this is a create of a record via a POST, I'm not sure why it is triggering an edit (requiring a record_id) which is also via a POST.

2 hours ago, BN_IT_Support said:

As a general opinion on "Pages DB"...

They are not really "databases" in SQL terms -- but SQL "tables" in the Invision database.

When you are considering which sort of SQL table to use (within Invision) you have two choices:

1. "Pages DB" (as you have mentioned)
2. SQL tables as defined under the 'Database Schema' tab for an Application

Pages DB works well for:

• Where you only need a single table for all your data - for example, in the classic database design if you only need a "products" table then it will work well. If you actually need a "products" table and a "suppliers" table with cross references then it does not work so well - and in any case you would need two Pages DB (one for products and one for suppliers)
   
• Where the solution involves a human adding and editing records it works well - for example, where you have a human adding products (descriptions, part numbers, prices, etc.) to a product table. We probably use between 5 and 10 Pages DB for various unrelated things such as "news articles", "places to go" and so on.
   
• Using Pages DB automatically gives you various widgets such as "Database filters" which is good.
   
• I would strongly recommend that you use the standard display templates wherever possible. If you create your own display templates based on the standard display templates then that effectively bases your template on a snapshot of the standard template - you don't benefit from future improvements and bug fixes to the standard. We have some of our own templates from 6 or 8 years ago and regret not putting more effort into display formatting for individual fields. With some clever display formats you can combine fields to produce complex output (for example, latitude and longitude fields to display a map) and if you need to do conditional display of some fields dependent upon the contents of others then you can add a dummy field (with a non-blank default value so it triggers display) and you can write custom code to display the contents of several other fields according to the rules that you want to apply. Do the fancy stuff in a field display rather than a template wherever possible.
   
• When you write a scheduled task (for example) to update the data in a Pages DB there are a couple of (minor) extra steps that you need to take in order to write the data. Firstly, you need to know the classname that will be used to write to the correct table and this will be something like \IPS\cms|\Records23 where 23 is the database number. So, you have to look up the database number from your database key (I recommend against hard coding the 23 😉 ). Secondly, if you have fields for name, address, region, country, phone, etc. then the names in the table will be something like field_91, field_92, field_93, field_94, etc so you have to 'map' from the field key to the real field name in the database. Not difficult, but that's what you have to do.
   
• Pages DB will work OK where you have a scheduled task to update the data once per day (for example - it could obviously be much more frequently if you wanted). A scheduled task will work fine where nearly all the data items/records are accessed every day - but what about a scenario where only 5% of records are accessed each day? Doing a daily scheduled update of all records means that 95% of what you retrieve is not going to be used in the next 24 hours. I rather liked Nathan's suggestion of retrieving data on demand (i.e. every record has a lifetime field that you add so if you want the record but the lifetime has expired then you retrieve the data through the API). That would mean that you would not waste resources retrieving data that is not being used. Unfortunately, "retrieval on demand" would be extremely difficult or impossible to implement using Pages DB - unless you write your own very fancy templates and in that case you could do better to write an Application to do the entire job. Updating Pages DB from a scheduled job would work easily.

Application tables will work well for:

• You need several tables with relationships between them (e.g. products and suppliers)
   
• You need an application that does a lot of processing of records (rather than simple human entry of records as indicated previously)
   
• Access to records is more intuitive - field names are what you want to call them rather than being 'mapped'. Also, access to the correct table is very simple as you need to create a sub-class of \IPS\Patterns\ActiveRecord that will get you to the correct table plus a whole load more stuff.
   
• If you have hierarchical data (for example, categories and records) then you should consider using the Node/Model/Item/Content model although that gets a lot more complicated it does give you access to many of the features that you see time and again in Invision.
   
• You have complete control so retrieving data on demand is much more simple than it would be using Pages DB.

Regards,

John

 

Awesome!  Thanks John for the very detailed reply and providing your experiences!  That's very helpful!